====== Search Code Engine ====== ===== Comparion Seach Code Engine Tools ===== Reference at https://github.com/OpenGrok/OpenGrok/wiki/Comparison-with-Similar-Tools ^Feature^LXR^ctags^cscope^ViewVC^GNU GLOBAL^OpenGrok^ |Full text Search| Y| | Y| | Y| Y| |Definition Search| P| Y| Y| | Y| Y| |Identifier Search| Y| | Y| | Y| Y| |Path search| Y| | Y| | Y| Y| |History Search| | | | | | Y| |Caller/Callee Search| | | Y| | P| | |RegExp Search| | | Y| | Y| P (only wildcards)| |Shows matching lines| | Y| Y| | Y| Y| |Hierarchical Search| | | | | Y| Y| |query syntax like AND, OR, field:| | | | | | Y| |Incremental update| | | P| | Y| Y| |RSS Feed| | | | Y| | Y| |Syntax highlighting-Xref| Y| | | P| Y| Y| |Interface for SCM| | | | Y| Y| Y| |open source| Y| Y| Y| Y| Y| Y| |Usable URLs| Y| N/A| N/A| | Y| Y| |Individual file download| | N/A| N/A| Y| N/A| Y| |Changes at directory level| | N/A| N/A| P| N/A| Y| |Multi language support| P| Y| P| N/A| Y| Y| |User interface| WEB| CLI| CLI| WEB| WEB / CLI| WEB / CLI| |Static HTML| | | | | Y| P (for xrefs)| |Input completion| | | | | Y| P (with greasemonkey script)| |Built-in parser| | Y| Y| | Y| P (lexical analysis)| |Plug-in parser| | | | | Y| | |Integration with (exuberant) ctags| | N/A| | | Y| Y| |Integration with cscope| | | N/A| | Y| | |Integration with emacs| | Y| Y| | Y| | |Integration with vim| | Y| Y| | Y| P (through https://github.com/jdevera/vim-opengrok-search)| |Integration with doxygen| | | | | Y| | |Integration with Netbeans| | | | | | | |Integration with less| | Y| | | Y| | |Integration with bash| | | | | Y| | |Integration with idutils| | | | | Y| | |Integration with Elvis| | Y| | | Y| And about language support, OpenGrok support the most programming languages:bzip2, C, C++, C#, ELF, Fortran, generic data, gzip, Java, Java archive, Java class, Java script, Lisp, Perl, PHP, plaintext, Python, Scala, Shell, SQL, PLSQL, tar, Tcl, troff, Visual basic, XML, zip ===== Ctags ===== Install Ctags: * Download Ctags at link http://ctags.sourceforge.net/ * Install tar xf ctags-5.8.tar.gz cd ctags-5.8 ./configure make make install ===== lxr ===== ==== Install and config lxr with Ctags ==== Get lxr at link http://sourceforge.net/projects/lxr/ === Pre-requisites === Run below command to check packages missing: ./genxref --checkonly * Install perl File:MMagic yum install perl-ExtUtils-MakeMaker wget http://www.cpan.org/authors/id/K/KN/KNOK/File-MMagic-1.30.tar.gz tar xf File-MMagic-1.30.tar.gz cd File-MMagic-1.30 perl perl Makefile.PL make & make install * Install perl Glimpse: yum install perl-Module-Build yum install perl-Test-Pod yum install perl-Test-Pod-Coverage wget http://webglimpse.net/trial/glimpse-latest.tar.gz tar xf glimpse-latest.tar.gz cd glimpse-4.18.6/ ./configure make & make install * Install mod_perl for apache: yum install perl-ExtUtils-Embed wget http://mirrors.digipower.vn/apache/perl/mod_perl-2.0.8.tar.gz tar xf mod_perl-2.0.8.tar.gz cd mod_perl-2.0.8 perl Makefile.PL MP_APXS=/usr/local/apache/bin/apxs make & make install add config mod_perl to /usr/local/apache/conf/httpd.conf:LoadModule perl_module modules/mod_perl.so * Install mod_version: /usr/local/apache/bin/httpd -M | grep version version_module (static) Syntax OK => mode_version has been installed === Install and Config lxr === - Install: mkdir -p /data/www/lxr cp -R lxr-2.0.2/* /data/www/lxr - Run script configure lxr with defauld configuration and below changes: ./scripts/configure-lxr.pl -vv Configure for single/multiple trees? [S/m] > m Server type? [dedicated/SHARED] > dedicated --- Host name or IP? [//localhost] > http://searchcode.babies.vn --- DB user name? [lxr] > --- DB password ? [lxrpw] > xxxxx --- Directory for glimpse databases? >/data/glimpsedirbase Is your Apache version 2.4 or higher? [YES/no] > no And config for first tree: --- Caption in page header? (e.g. Project XYZZY displayed by LXR) > python --- Short title for button? (e.g. XYZZY) > python --- Tree identification in URL? (e.g. the-tree) > python --- Source directory? (e.g. /home/myself/project-tree) > /data/code --- Database name? > python --- Version name? > 2.7 *** Configure another tree? [YES/no] > no - prepare config and check cp custom.d/lxr.conf ./genxref --checkonly Output: [ OK ] Perl version ... 5.10.1 [ OK ] ctags version ... 5.8 Checked: glimpse version ... 4.18.5 Checked: glimpseindex version ... 4.18.5 Parameter 'swishbin' not defined - trying to find swish-e swish-e not found, `command -v swish-e` returned a null string genxref stopped without indexing by --checkonly option - Edit config lxr.conf: 'glimpsedirbase' => '/data/glimpsedirbase' 'sourceroot' => '/data/code' 'range' => [qw(python2.7)] And below is new config: ( { 'routing' => 'argument' , 'tmpdir' => '/tmp' , 'glimpsebin' => '/usr/local/bin/glimpse' , 'glimpseindex' => '/usr/local/bin/glimpseindex' , 'glimpsedirbase' => '/data/glimpsedirbase' , 'swishconf' => '/data/www/lxr/templates/swish-e.conf' , 'ectagsbin' => '/usr/local/bin/ctags' , 'ectagsconf' => '/data/www/lxr/templates/ectags.conf' , 'cvspath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin' , 'gitpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin' , 'hgpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin' , 'svnpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin' , 'host_names' => [ 'http://searchcode.babies.vn' ] , 'htmlfatal' => '/data/www/lxr/templates/html/html-fatal.html' , 'htmlhead' => '/data/www/lxr/templates/html/html-head-btn.html' , 'htmltail' => '/data/www/lxr/templates/html/html-tail.html' , 'htmldir' => '/data/www/lxr/templates/html/html-dir-indexing.html' , 'htmlident' => '/data/www/lxr/templates/html/html-ident.html' , 'htmlsearch' => '/data/www/lxr/templates/html/html-search-glimpse.html' , 'htmlconfig' => '/data/www/lxr/templates/html/html-config.html' , 'showconfighead' => '/data/www/lxr/templates/html/config-head-btn-smaller.html' , 'diffleftwidth' => 50 , 'stylesheet' => 'templates/lxr.css' , 'alternate_stylesheet' => [ 'templates/classic.css' ] , 'encoding' => 'utf-8' , 'ignorefiles' => '^\\.|~$|\\.(o|a|orig)$|^CVS$|^core$' , 'graphicfile' => 'bitmap|bmp|gif|icon?|jp2|jpe?g|pjpe?g|png|svg|tiff?|xbm|xpm' , 'filetypeconf' => '/data/www/lxr/templates/filetype.conf' , 'genericconf' => '/data/www/lxr/lib/LXR/Lang/generic.conf' , 'dbuser' => 'lxr' , 'dbpass' => 'xxxxx' , 'dbprefix' => 'lxr_' } , { 'virtroot' => '' , 'caption' => 'python' , 'shortcaption' => 'python' , 'treename' => 'python' , 'sourceroot' => '/data/code/python' , 'sourcerootname' => '$v' , 'variables' => { 'v' => { 'name' => 'Version' , 'range' => [qw( 2.7 )] } } , 'dbname' => 'dbi:mysql:dbname=python' } ) - Run script create new mysql user for tree and init database: ./custom.d/initdb.sh - Re-grant sql access for lxr user: grant all privileges on python.* to 'lxr'@'localhost' identified by 'xxxxxx'; - Generate Index ./genxref --url=http://searchcode.babies.vn --tree=python --version=2.7 === config apache for running first tree source === NameVirtualHost *:80 DocumentRoot /data/www/lxr ServerName searchcode.babies.vn PerlSwitches -T PerlTaintCheck On PerlPostConfigRequire /data/www/lxr/custom.d/apache2-require.pl PerlRequire /data/www/lxr/custom.d/apache2-require.pl Options FollowSymLinks AllowOverride AuthConfig FileInfo Limit Options Order allow,deny Allow from all ==== Create and config new tree source with perl script ==== * Run script create new tree source with below change configuration: ./scripts/configure-lxr.pl --add --- Caption in page header? (e.g. Project XYZZY displayed by LXR) > drupal --- Short title for button? (e.g. XYZZY) > drupal --- Tree identification in URL? (e.g. the-tree) > drupal --- Source directory? (e.g. /home/myself/project-tree) > /data/code/drupal --- Version name? > 7.28 * Update configuration:Copy the augmented lxr.conf configuration file and physically create the database: cp ./custom.d/lxr.conf . ./custom.d/initdb.sh * Index your trees: Run the genxref script for each tree ./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28 ==== Create and config new tree source no perl script ==== * grant for user lxr access database drupal: GRANT ALL privileges ON drupal.* TO 'lxr'@'localhost'; * create perl script initdb_drupal.sh: echo "*** MySQL - Creating tree database drupal" mysql -u lxr -plxr12!@ < 0; /* Types for a language */ /* declaration: provided by generic.conf */ create table lxr_langtypes ( typeid smallint not null , langid tinyint unsigned not null , declaration varchar(255) not null , constraint lxr_pk_langtypes primary key (typeid, langid) ) engine = MyISAM; /* Symbol name dictionary */ /* symid: unique symbol id for name * symcount: number of definitions and usages for this name * symname: symbol name */ create table lxr_symbols ( symid int not null primary key , symcount int , symname varbinary(255) not null unique ) engine = MyISAM; /* The following function decrements the symbol reference count * (to be used in triggers). */ delimiter // create procedure lxr_decsym(in whichsym int) begin update lxr_symbols set symcount = symcount - 1 where symid = whichsym and symcount > 0; end// delimiter ; /* Definitions */ /* symid: refers to symbol name * fileid and line define the location of the declaration * langid: language id * typeid: language type id * relid: optional id of the englobing declaration * (refers to another symbol, not a definition) */ create table lxr_definitions ( symid int not null , fileid int not null , line int not null , typeid smallint not null , langid tinyint unsigned not null , relid int , index lxr_i_definitions (symid) , constraint lxr_fk_defn_symid foreign key (symid) references lxr_symbols(symid) , constraint lxr_fk_defn_fileid foreign key (fileid) references lxr_files(fileid) , constraint lxr_fk_defn_type foreign key (typeid, langid) references lxr_langtypes(typeid, langid) , constraint lxr_fk_defn_relid foreign key (relid) references lxr_symbols(symid) ) engine = MyISAM; /* The following trigger maintains correct symbol reference count * after definition deletion. */ delimiter // drop trigger if exists lxr_remove_definition; create trigger lxr_remove_definition after delete on lxr_definitions for each row begin call lxr_decsym(old.symid); if old.relid is not null then call lxr_decsym(old.relid); end if; end// delimiter ; /* Usages */ create table lxr_usages ( symid int not null , fileid int not null , line int not null , index lxr_i_usages (symid) , constraint lxr_fk_use_symid foreign key (symid) references lxr_symbols(symid) , constraint lxr_fk_use_fileid foreign key (fileid) references lxr_files(fileid) ) engine = MyISAM; /* The following trigger maintains correct symbol reference count * after usage deletion. */ drop trigger if exists lxr_remove_usage; create trigger lxr_remove_usage after delete on lxr_usages for each row call lxr_decsym(old.symid); delimiter // create procedure lxr_PurgeAll () begin set @old_check = @@session.foreign_key_checks; set session foreign_key_checks = OFF; truncate table lxr_filenum; truncate table lxr_symnum; truncate table lxr_typenum; insert into lxr_filenum (rcd, fid) VALUES (0, 0); insert into lxr_symnum (rcd, sid) VALUES (0, 0); insert into lxr_typenum (rcd, tid) VALUES (0, 0); truncate table lxr_definitions; truncate table lxr_usages; truncate table lxr_langtypes; truncate table lxr_symbols; truncate table lxr_releases; truncate table lxr_status; truncate table lxr_files; set session foreign_key_checks = @old_check; end// delimiter ; END_OF_TEMPLATE * add config for drupal in lxr.conf: ,{ 'virtroot' => '' , 'caption' => 'drupal' , 'shortcaption' => 'drupal' , 'treename' => 'drupal' , 'sourceroot' => '/data/code/drupal' , 'sourcerootname' => '$v' , 'variables' => { 'v' => { 'name' => 'Version' , 'range' => [qw( 7.28 )] } } , 'dbname' => 'dbi:mysql:dbname=drupal' } * run Index for drupal: ./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28 ===== OpenGrok ===== {{:other:setup-project.png|}} ==== Requirements for Install ==== Requirements: * [[http://www.oracle.com/technetwork/java/|JDK 1.7]] or higher * [[searchcodeengine#ctags|Exuberant Ctags]] for analysis * Webserver [[http://tomcat.apache.org/|Tomcat]] Install requirements on CentOS: yum install java-1.7.0-openjdk yum install tomcat6 ==== Install on CentOS ==== wget https://java.net/projects/opengrok/downloads/download/opengrok-0.12.1.tar.gz tar xf opengrok-0.12.1.tar.gz cp -R opengrok-0.12.1 /usr/local ==== Index and install web application opengrok ==== - Deploy the web application cd /usr/local/opengrok/bin ./OpenGrok deploy => web opengrok will be installed to **/var/lib/tomcat6/webapps/source.war** or **/usr/share/tomcat6/webapps/source.war** - Remove unused source in /data/code and run Index to generate project config files: cd /data/code for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -f $i; done done And run Index: cd /usr/local/opengrok-0.12.1/bin/ ./OpenGrok index /data/code/ => Generate **/var/opengrok** and output: 00:03:41 SEVERE: Failed to send configuration to localhost:2424 (is web application server running with opengrok deployed?) java.net.ConnectException: Connection refused .................. Port 2424 was configured in webapp **/var/lib/tomcat6/webapps/source.war/WEB-INF/web.xml** - Check search code after index: java -cp ./opengrok.jar org.opensolaris.opengrok.search.Search -R /var/opengrok/etc/configuration.xml -f crawler.py => Display search result for string "crawler.py" - Restart tomcat6: /etc/init.d/tomcat6 restart => After restart, webapp OpenGrok will start and **listen on Port 2424** for java application OpenGrok.jar connet and transfer data after run Index ==== Config Tomcat and check ==== === Understand directory structuer and environment in tomcat6 === Directory structure of tomcat6 on centos:**/usr/share/tomcat6/** drwxr-xr-x 2 root root 4096 Jun 14 09:43 bin lrwxrwxrwx 1 root tomcat 12 Jun 14 09:43 conf -> /etc/tomcat6 lrwxrwxrwx 1 root root 23 Jun 14 09:43 lib -> /usr/share/java/tomcat6 lrwxrwxrwx 1 root root 16 Jun 14 09:43 logs -> /var/log/tomcat6 lrwxrwxrwx 1 root root 23 Jun 14 09:43 temp -> /var/cache/tomcat6/temp lrwxrwxrwx 1 root root 24 Jun 14 09:43 webapps -> /var/lib/tomcat6/webapps lrwxrwxrwx 1 root root 23 Jun 14 09:43 work -> /var/cache/tomcat6/work And environment variables for tomcat server: ps -eaf | grep tomcat => output: /usr/lib/jvm/jre/bin/java -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -classpath :/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat6/temp -Djava.util.logging.config.file=/usr/share/tomcat6/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start === Check search code === goto link:http://128.199.236.122:8080/source/ ==== Rebuild OpenGrok from source ==== === Requirements === Requirements: * JDK 1.7 or higher * Apache Ant 1.8 or later Prepare requirements on centos: * Install JDK 1.7: yum install java-1.7.0-openjdk yum install java-1.7.0-openjdk-devel * Install Ant 1.8 wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.gz tar xf apache-ant-1.8.0-bin.tar.gz cp -R apache-ant-1.8.0 /usr/local === Rebuild OpenGrok === * Prepare environment: export JAVA_HOME=/usr/lib/jvm/java-1.7.0 export PATH=$PATH:/usr/local/apache-ant-1.8.0/bin/ * Build: wget https://github.com/OpenGrok/OpenGrok/archive/0.12.1.tar.gz tar xf 0.12.1.tar.gz cd OpenGrok-0.12.1 ant => output: ................... -copy-lib-without-netbeans: [copy] Copying 1 file to /root/source/searchcodeengine/OpenGrok-0.12.1/dist/lib [copy] Copying 8 files to /root/source/searchcodeengine/OpenGrok-0.12.1/dist -post-jar: [war] Building war: /root/source/searchcodeengine/OpenGrok-0.12.1/dist/source.war [echo] Generating man page.. jar: BUILD SUCCESSFUL Total time: 16 seconds === Edit for recognizing drupal file === Go to directory **OpenGrok-0.12.1/src/org/opensolaris/opengrok/analysis/php** ==== Reindex source ==== - Update new source code to /data/code, for example python2.7: rm -rf /data/code/python2.7 cp -R /usr/local/lib/python2.7/ /data/code/ - Remove code unused: cd /data/code for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -rf $i; done done - reindex the /data/code cd /usr/local/opengrok-0.12.1/bin/ ./OpenGrok index /data/code/