====== Search Code Engine ======
===== Comparion Seach Code Engine Tools =====
Reference at https://github.com/OpenGrok/OpenGrok/wiki/Comparison-with-Similar-Tools
^Feature^LXR^ctags^cscope^ViewVC^GNU GLOBAL^OpenGrok^
|Full text Search| Y| | Y| | Y| Y|
|Definition Search| P| Y| Y| | Y| Y|
|Identifier Search| Y| | Y| | Y| Y|
|Path search| Y| | Y| | Y| Y|
|History Search| | | | | | Y|
|Caller/Callee Search| | | Y| | P| |
|RegExp Search| | | Y| | Y| P (only wildcards)|
|Shows matching lines| | Y| Y| | Y| Y|
|Hierarchical Search| | | | | Y| Y|
|query syntax like AND, OR, field:| | | | | | Y|
|Incremental update| | | P| | Y| Y|
|RSS Feed| | | | Y| | Y|
|Syntax highlighting-Xref| Y| | | P| Y| Y|
|Interface for SCM| | | | Y| Y| Y|
|open source| Y| Y| Y| Y| Y| Y|
|Usable URLs| Y| N/A| N/A| | Y| Y|
|Individual file download| | N/A| N/A| Y| N/A| Y|
|Changes at directory level| | N/A| N/A| P| N/A| Y|
|Multi language support| P| Y| P| N/A| Y| Y|
|User interface| WEB| CLI| CLI| WEB| WEB / CLI| WEB / CLI|
|Static HTML| | | | | Y| P (for xrefs)|
|Input completion| | | | | Y| P (with greasemonkey script)|
|Built-in parser| | Y| Y| | Y| P (lexical analysis)|
|Plug-in parser| | | | | Y| |
|Integration with (exuberant) ctags| | N/A| | | Y| Y|
|Integration with cscope| | | N/A| | Y| |
|Integration with emacs| | Y| Y| | Y| |
|Integration with vim| | Y| Y| | Y| P (through https://github.com/jdevera/vim-opengrok-search)|
|Integration with doxygen| | | | | Y| |
|Integration with Netbeans| | | | | | |
|Integration with less| | Y| | | Y| |
|Integration with bash| | | | | Y| |
|Integration with idutils| | | | | Y| |
|Integration with Elvis| | Y| | | Y|
And about language support, OpenGrok support the most programming languages:bzip2, C, C++, C#, ELF, Fortran, generic data, gzip, Java, Java archive, Java class, Java script, Lisp, Perl, PHP, plaintext, Python, Scala, Shell, SQL, PLSQL, tar, Tcl, troff, Visual basic, XML, zip
===== Ctags =====
Install Ctags:
* Download Ctags at link http://ctags.sourceforge.net/
* Install
tar xf ctags-5.8.tar.gz
cd ctags-5.8
./configure
make
make install
===== lxr =====
==== Install and config lxr with Ctags ====
Get lxr at link http://sourceforge.net/projects/lxr/
=== Pre-requisites ===
Run below command to check packages missing:
./genxref --checkonly
* Install perl File:MMagic
yum install perl-ExtUtils-MakeMaker
wget http://www.cpan.org/authors/id/K/KN/KNOK/File-MMagic-1.30.tar.gz
tar xf File-MMagic-1.30.tar.gz
cd File-MMagic-1.30
perl perl Makefile.PL
make & make install
* Install perl Glimpse:
yum install perl-Module-Build
yum install perl-Test-Pod
yum install perl-Test-Pod-Coverage
wget http://webglimpse.net/trial/glimpse-latest.tar.gz
tar xf glimpse-latest.tar.gz
cd glimpse-4.18.6/
./configure
make & make install
* Install mod_perl for apache:
yum install perl-ExtUtils-Embed
wget http://mirrors.digipower.vn/apache/perl/mod_perl-2.0.8.tar.gz
tar xf mod_perl-2.0.8.tar.gz
cd mod_perl-2.0.8
perl Makefile.PL MP_APXS=/usr/local/apache/bin/apxs
make & make install
add config mod_perl to /usr/local/apache/conf/httpd.conf:LoadModule perl_module modules/mod_perl.so
* Install mod_version:
/usr/local/apache/bin/httpd -M | grep version
version_module (static)
Syntax OK
=> mode_version has been installed
=== Install and Config lxr ===
- Install:
mkdir -p /data/www/lxr
cp -R lxr-2.0.2/* /data/www/lxr
- Run script configure lxr with defauld configuration and below changes:
./scripts/configure-lxr.pl -vv
Configure for single/multiple trees? [S/m] > m
Server type? [dedicated/SHARED] > dedicated
--- Host name or IP? [//localhost] > http://searchcode.babies.vn
--- DB user name? [lxr] >
--- DB password ? [lxrpw] > xxxxx
--- Directory for glimpse databases? >/data/glimpsedirbase
Is your Apache version 2.4 or higher? [YES/no] > no
And config for first tree:
--- Caption in page header? (e.g. Project XYZZY displayed by LXR) > python
--- Short title for button? (e.g. XYZZY) > python
--- Tree identification in URL? (e.g. the-tree) > python
--- Source directory? (e.g. /home/myself/project-tree) > /data/code
--- Database name? > python
--- Version name? > 2.7
*** Configure another tree? [YES/no] > no
- prepare config and check
cp custom.d/lxr.conf
./genxref --checkonly
Output:
[ OK ] Perl version ... 5.10.1
[ OK ] ctags version ... 5.8
Checked: glimpse version ... 4.18.5
Checked: glimpseindex version ... 4.18.5
Parameter 'swishbin' not defined - trying to find swish-e
swish-e not found, `command -v swish-e` returned a null string
genxref stopped without indexing by --checkonly option
- Edit config lxr.conf:
'glimpsedirbase' => '/data/glimpsedirbase'
'sourceroot' => '/data/code'
'range' => [qw(python2.7)]
And below is new config:
(
{
'routing' => 'argument'
, 'tmpdir' => '/tmp'
, 'glimpsebin' => '/usr/local/bin/glimpse'
, 'glimpseindex' => '/usr/local/bin/glimpseindex'
, 'glimpsedirbase' => '/data/glimpsedirbase'
, 'swishconf' => '/data/www/lxr/templates/swish-e.conf'
, 'ectagsbin' => '/usr/local/bin/ctags'
, 'ectagsconf' => '/data/www/lxr/templates/ectags.conf'
, 'cvspath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
, 'gitpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
, 'hgpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
, 'svnpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
, 'host_names' => [ 'http://searchcode.babies.vn'
]
, 'htmlfatal' => '/data/www/lxr/templates/html/html-fatal.html'
, 'htmlhead' => '/data/www/lxr/templates/html/html-head-btn.html'
, 'htmltail' => '/data/www/lxr/templates/html/html-tail.html'
, 'htmldir' => '/data/www/lxr/templates/html/html-dir-indexing.html'
, 'htmlident' => '/data/www/lxr/templates/html/html-ident.html'
, 'htmlsearch' => '/data/www/lxr/templates/html/html-search-glimpse.html'
, 'htmlconfig' => '/data/www/lxr/templates/html/html-config.html'
, 'showconfighead' => '/data/www/lxr/templates/html/config-head-btn-smaller.html'
, 'diffleftwidth' => 50
, 'stylesheet' => 'templates/lxr.css'
, 'alternate_stylesheet' => [ 'templates/classic.css' ]
, 'encoding' => 'utf-8'
, 'ignorefiles' =>
'^\\.|~$|\\.(o|a|orig)$|^CVS$|^core$'
, 'graphicfile' =>
'bitmap|bmp|gif|icon?|jp2|jpe?g|pjpe?g|png|svg|tiff?|xbm|xpm'
, 'filetypeconf' => '/data/www/lxr/templates/filetype.conf'
, 'genericconf' => '/data/www/lxr/lib/LXR/Lang/generic.conf'
, 'dbuser' => 'lxr'
, 'dbpass' => 'xxxxx'
, 'dbprefix' => 'lxr_'
}
, {
'virtroot' => ''
,
'caption' => 'python'
, 'shortcaption' => 'python'
, 'treename' => 'python'
, 'sourceroot' => '/data/code/python'
, 'sourcerootname' => '$v'
, 'variables' =>
{ 'v' =>
{ 'name' => 'Version'
, 'range' => [qw(
2.7
)]
}
}
, 'dbname' => 'dbi:mysql:dbname=python'
}
)
- Run script create new mysql user for tree and init database:
./custom.d/initdb.sh
- Re-grant sql access for lxr user:
grant all privileges on python.* to 'lxr'@'localhost' identified by 'xxxxxx';
- Generate Index
./genxref --url=http://searchcode.babies.vn --tree=python --version=2.7
=== config apache for running first tree source ===
NameVirtualHost *:80
DocumentRoot /data/www/lxr
ServerName searchcode.babies.vn
PerlSwitches -T
PerlTaintCheck On
PerlPostConfigRequire /data/www/lxr/custom.d/apache2-require.pl
PerlRequire /data/www/lxr/custom.d/apache2-require.pl
Options FollowSymLinks
AllowOverride AuthConfig FileInfo Limit Options
Order allow,deny
Allow from all
==== Create and config new tree source with perl script ====
* Run script create new tree source with below change configuration:
./scripts/configure-lxr.pl --add
--- Caption in page header? (e.g. Project XYZZY displayed by LXR) > drupal
--- Short title for button? (e.g. XYZZY) > drupal
--- Tree identification in URL? (e.g. the-tree) > drupal
--- Source directory? (e.g. /home/myself/project-tree) > /data/code/drupal
--- Version name? > 7.28
* Update configuration:Copy the augmented lxr.conf configuration file and physically create the database:
cp ./custom.d/lxr.conf .
./custom.d/initdb.sh
* Index your trees: Run the genxref script for each tree
./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28
==== Create and config new tree source no perl script ====
* grant for user lxr access database drupal:
GRANT ALL privileges ON drupal.* TO 'lxr'@'localhost';
* create perl script initdb_drupal.sh:
echo "*** MySQL - Creating tree database drupal"
mysql -u lxr -plxr12!@ < 0;
/* Types for a language */
/* declaration: provided by generic.conf
*/
create table lxr_langtypes
( typeid smallint not null
, langid tinyint unsigned not null
, declaration varchar(255) not null
, constraint lxr_pk_langtypes
primary key (typeid, langid)
)
engine = MyISAM;
/* Symbol name dictionary */
/* symid: unique symbol id for name
* symcount: number of definitions and usages for this name
* symname: symbol name
*/
create table lxr_symbols
( symid int not null primary key
, symcount int
, symname varbinary(255) not null unique
)
engine = MyISAM;
/* The following function decrements the symbol reference count
* (to be used in triggers).
*/
delimiter //
create procedure lxr_decsym(in whichsym int)
begin
update lxr_symbols
set symcount = symcount - 1
where symid = whichsym
and symcount > 0;
end//
delimiter ;
/* Definitions */
/* symid: refers to symbol name
* fileid and line define the location of the declaration
* langid: language id
* typeid: language type id
* relid: optional id of the englobing declaration
* (refers to another symbol, not a definition)
*/
create table lxr_definitions
( symid int not null
, fileid int not null
, line int not null
, typeid smallint not null
, langid tinyint unsigned not null
, relid int
, index lxr_i_definitions (symid)
, constraint lxr_fk_defn_symid
foreign key (symid)
references lxr_symbols(symid)
, constraint lxr_fk_defn_fileid
foreign key (fileid)
references lxr_files(fileid)
, constraint lxr_fk_defn_type
foreign key (typeid, langid)
references lxr_langtypes(typeid, langid)
, constraint lxr_fk_defn_relid
foreign key (relid)
references lxr_symbols(symid)
)
engine = MyISAM;
/* The following trigger maintains correct symbol reference count
* after definition deletion.
*/
delimiter //
drop trigger if exists lxr_remove_definition;
create trigger lxr_remove_definition
after delete on lxr_definitions
for each row
begin
call lxr_decsym(old.symid);
if old.relid is not null
then call lxr_decsym(old.relid);
end if;
end//
delimiter ;
/* Usages */
create table lxr_usages
( symid int not null
, fileid int not null
, line int not null
, index lxr_i_usages (symid)
, constraint lxr_fk_use_symid
foreign key (symid)
references lxr_symbols(symid)
, constraint lxr_fk_use_fileid
foreign key (fileid)
references lxr_files(fileid)
)
engine = MyISAM;
/* The following trigger maintains correct symbol reference count
* after usage deletion.
*/
drop trigger if exists lxr_remove_usage;
create trigger lxr_remove_usage
after delete on lxr_usages
for each row
call lxr_decsym(old.symid);
delimiter //
create procedure lxr_PurgeAll ()
begin
set @old_check = @@session.foreign_key_checks;
set session foreign_key_checks = OFF;
truncate table lxr_filenum;
truncate table lxr_symnum;
truncate table lxr_typenum;
insert into lxr_filenum
(rcd, fid) VALUES (0, 0);
insert into lxr_symnum
(rcd, sid) VALUES (0, 0);
insert into lxr_typenum
(rcd, tid) VALUES (0, 0);
truncate table lxr_definitions;
truncate table lxr_usages;
truncate table lxr_langtypes;
truncate table lxr_symbols;
truncate table lxr_releases;
truncate table lxr_status;
truncate table lxr_files;
set session foreign_key_checks = @old_check;
end//
delimiter ;
END_OF_TEMPLATE
* add config for drupal in lxr.conf:
,{
'virtroot' => ''
, 'caption' => 'drupal'
, 'shortcaption' => 'drupal'
, 'treename' => 'drupal'
, 'sourceroot' => '/data/code/drupal'
, 'sourcerootname' => '$v'
, 'variables' =>
{ 'v' =>
{ 'name' => 'Version'
, 'range' => [qw(
7.28
)]
}
}
, 'dbname' => 'dbi:mysql:dbname=drupal'
}
* run Index for drupal:
./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28
===== OpenGrok =====
{{:other:setup-project.png|}}
==== Requirements for Install ====
Requirements:
* [[http://www.oracle.com/technetwork/java/|JDK 1.7]] or higher
* [[searchcodeengine#ctags|Exuberant Ctags]] for analysis
* Webserver [[http://tomcat.apache.org/|Tomcat]]
Install requirements on CentOS:
yum install java-1.7.0-openjdk
yum install tomcat6
==== Install on CentOS ====
wget https://java.net/projects/opengrok/downloads/download/opengrok-0.12.1.tar.gz
tar xf opengrok-0.12.1.tar.gz
cp -R opengrok-0.12.1 /usr/local
==== Index and install web application opengrok ====
- Deploy the web application
cd /usr/local/opengrok/bin
./OpenGrok deploy
=> web opengrok will be installed to **/var/lib/tomcat6/webapps/source.war** or **/usr/share/tomcat6/webapps/source.war**
- Remove unused source in /data/code and run Index to generate project config files:
cd /data/code
for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done
for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -f $i; done done
And run Index:
cd /usr/local/opengrok-0.12.1/bin/
./OpenGrok index /data/code/
=> Generate **/var/opengrok** and output:
00:03:41 SEVERE: Failed to send configuration to localhost:2424 (is web application server running with opengrok deployed?)
java.net.ConnectException: Connection refused
..................
Port 2424 was configured in webapp **/var/lib/tomcat6/webapps/source.war/WEB-INF/web.xml**
- Check search code after index:
java -cp ./opengrok.jar org.opensolaris.opengrok.search.Search -R /var/opengrok/etc/configuration.xml -f crawler.py
=> Display search result for string "crawler.py"
- Restart tomcat6:
/etc/init.d/tomcat6 restart
=> After restart, webapp OpenGrok will start and **listen on Port 2424** for java application OpenGrok.jar connet and transfer data after run Index
==== Config Tomcat and check ====
=== Understand directory structuer and environment in tomcat6 ===
Directory structure of tomcat6 on centos:**/usr/share/tomcat6/**
drwxr-xr-x 2 root root 4096 Jun 14 09:43 bin
lrwxrwxrwx 1 root tomcat 12 Jun 14 09:43 conf -> /etc/tomcat6
lrwxrwxrwx 1 root root 23 Jun 14 09:43 lib -> /usr/share/java/tomcat6
lrwxrwxrwx 1 root root 16 Jun 14 09:43 logs -> /var/log/tomcat6
lrwxrwxrwx 1 root root 23 Jun 14 09:43 temp -> /var/cache/tomcat6/temp
lrwxrwxrwx 1 root root 24 Jun 14 09:43 webapps -> /var/lib/tomcat6/webapps
lrwxrwxrwx 1 root root 23 Jun 14 09:43 work -> /var/cache/tomcat6/work
And environment variables for tomcat server:
ps -eaf | grep tomcat
=> output:
/usr/lib/jvm/jre/bin/java
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory
-classpath :/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
-Dcatalina.base=/usr/share/tomcat6
-Dcatalina.home=/usr/share/tomcat6
-Djava.endorsed.dirs=
-Djava.io.tmpdir=/var/cache/tomcat6/temp
-Djava.util.logging.config.file=/usr/share/tomcat6/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap
start
=== Check search code ===
goto link:http://128.199.236.122:8080/source/
==== Rebuild OpenGrok from source ====
=== Requirements ===
Requirements:
* JDK 1.7 or higher
* Apache Ant 1.8 or later
Prepare requirements on centos:
* Install JDK 1.7:
yum install java-1.7.0-openjdk
yum install java-1.7.0-openjdk-devel
* Install Ant 1.8
wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.gz
tar xf apache-ant-1.8.0-bin.tar.gz
cp -R apache-ant-1.8.0 /usr/local
=== Rebuild OpenGrok ===
* Prepare environment:
export JAVA_HOME=/usr/lib/jvm/java-1.7.0
export PATH=$PATH:/usr/local/apache-ant-1.8.0/bin/
* Build:
wget https://github.com/OpenGrok/OpenGrok/archive/0.12.1.tar.gz
tar xf 0.12.1.tar.gz
cd OpenGrok-0.12.1
ant
=> output:
...................
-copy-lib-without-netbeans:
[copy] Copying 1 file to /root/source/searchcodeengine/OpenGrok-0.12.1/dist/lib
[copy] Copying 8 files to /root/source/searchcodeengine/OpenGrok-0.12.1/dist
-post-jar:
[war] Building war: /root/source/searchcodeengine/OpenGrok-0.12.1/dist/source.war
[echo] Generating man page..
jar:
BUILD SUCCESSFUL
Total time: 16 seconds
=== Edit for recognizing drupal file ===
Go to directory **OpenGrok-0.12.1/src/org/opensolaris/opengrok/analysis/php**
==== Reindex source ====
- Update new source code to /data/code, for example python2.7:
rm -rf /data/code/python2.7
cp -R /usr/local/lib/python2.7/ /data/code/
- Remove code unused:
cd /data/code
for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done
for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -rf $i; done done
- reindex the /data/code
cd /usr/local/opengrok-0.12.1/bin/
./OpenGrok index /data/code/