User Tools

Site Tools


searchcodeengine

Search Code Engine

Comparion Seach Code Engine Tools

Reference at https://github.com/OpenGrok/OpenGrok/wiki/Comparison-with-Similar-Tools

FeatureLXRctagscscopeViewVCGNU GLOBALOpenGrok
Full text Search Y Y Y Y
Definition Search P Y Y Y Y
Identifier Search Y Y Y Y
Path search Y Y Y Y
History Search Y
Caller/Callee Search Y P
RegExp Search Y Y P (only wildcards)
Shows matching lines Y Y Y Y
Hierarchical Search Y Y
query syntax like AND, OR, field: Y
Incremental update P Y Y
RSS Feed Y Y
Syntax highlighting-Xref Y P Y Y
Interface for SCM Y Y Y
open source Y Y Y Y Y Y
Usable URLs Y N/A N/A Y Y
Individual file download N/A N/A Y N/A Y
Changes at directory level N/A N/A P N/A Y
Multi language support P Y P N/A Y Y
User interface WEB CLI CLI WEB WEB / CLI WEB / CLI
Static HTML Y P (for xrefs)
Input completion Y P (with greasemonkey script)
Built-in parser Y Y Y P (lexical analysis)
Plug-in parser Y
Integration with (exuberant) ctags N/A Y Y
Integration with cscope N/A Y
Integration with emacs Y Y Y
Integration with vim Y Y Y P (through https://github.com/jdevera/vim-opengrok-search)
Integration with doxygen Y
Integration with Netbeans
Integration with less Y Y
Integration with bash Y
Integration with idutils Y
Integration with Elvis Y Y

And about language support, OpenGrok support the most programming languages:bzip2, C, C++, C#, ELF, Fortran, generic data, gzip, Java, Java archive, Java class, Java script, Lisp, Perl, PHP, plaintext, Python, Scala, Shell, SQL, PLSQL, tar, Tcl, troff, Visual basic, XML, zip

Ctags

Install Ctags:

lxr

Install and config lxr with Ctags

Pre-requisites

Run below command to check packages missing:

./genxref --checkonly
  • Install perl File:MMagic
    yum install perl-ExtUtils-MakeMaker
    wget http://www.cpan.org/authors/id/K/KN/KNOK/File-MMagic-1.30.tar.gz
    tar xf File-MMagic-1.30.tar.gz
    cd File-MMagic-1.30
    perl perl Makefile.PL
    make & make install
  • Install perl Glimpse:
    yum install perl-Module-Build
    yum install perl-Test-Pod
    yum install perl-Test-Pod-Coverage
    wget http://webglimpse.net/trial/glimpse-latest.tar.gz
    tar xf glimpse-latest.tar.gz
    cd glimpse-4.18.6/
    ./configure
    make & make install
  • Install mod_perl for apache:
    yum install perl-ExtUtils-Embed
    wget http://mirrors.digipower.vn/apache/perl/mod_perl-2.0.8.tar.gz
    tar xf mod_perl-2.0.8.tar.gz
    cd mod_perl-2.0.8
    perl Makefile.PL MP_APXS=/usr/local/apache/bin/apxs
    make & make install

    add config mod_perl to /usr/local/apache/conf/httpd.conf:

    LoadModule perl_module modules/mod_perl.so
  • Install mod_version:
    /usr/local/apache/bin/httpd -M | grep version
    version_module (static)
    Syntax OK

    ⇒ mode_version has been installed

Install and Config lxr

  1. Install:
    mkdir -p /data/www/lxr
    cp -R lxr-2.0.2/* /data/www/lxr
  2. Run script configure lxr with defauld configuration and below changes:
    ./scripts/configure-lxr.pl -vv
    Configure for single/multiple trees? [S/m] > m
    Server type? [dedicated/SHARED] > dedicated
    --- Host name or IP? [//localhost] > http://searchcode.babies.vn
    --- DB user name? [lxr] >
    --- DB password ? [lxrpw] > xxxxx
    --- Directory for glimpse databases? >/data/glimpsedirbase
    Is your Apache version 2.4 or higher? [YES/no] > no

    And config for first tree:

    --- Caption in page header? (e.g. Project XYZZY displayed by LXR) > python
    --- Short title for button? (e.g. XYZZY) > python
    --- Tree identification in URL? (e.g. the-tree) > python
    --- Source directory? (e.g. /home/myself/project-tree) > /data/code
    --- Database name? > python
    --- Version name?  > 2.7
    *** Configure another tree? [YES/no] > no
  3. prepare config and check
    cp custom.d/lxr.conf
    ./genxref --checkonly

    Output:

    [  OK  ]     Perl     version ... 5.10.1
    [  OK  ]     ctags    version ... 5.8
    Checked:    glimpse   version ... 4.18.5
    Checked: glimpseindex version ... 4.18.5
    Parameter 'swishbin' not defined - trying to find swish-e
    swish-e not found, `command -v swish-e` returned a null string
    genxref stopped without indexing by --checkonly option
  4. Edit config lxr.conf:
    'glimpsedirbase' => '/data/glimpsedirbase'
    'sourceroot' => '/data/code'
     'range' => [qw(python2.7)]

    And below is new config:

    (
            {
              'routing' => 'argument'
            , 'tmpdir' => '/tmp'
            , 'glimpsebin'     => '/usr/local/bin/glimpse'
            , 'glimpseindex'   => '/usr/local/bin/glimpseindex'
            , 'glimpsedirbase' => '/data/glimpsedirbase'
            , 'swishconf' => '/data/www/lxr/templates/swish-e.conf'
            , 'ectagsbin' => '/usr/local/bin/ctags'
            , 'ectagsconf' => '/data/www/lxr/templates/ectags.conf'
            , 'cvspath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
            , 'gitpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
            , 'hgpath'  => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
            , 'svnpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
            ,       'host_names' => [ 'http://searchcode.babies.vn'
                                                    ]
            , 'htmlfatal'           => '/data/www/lxr/templates/html/html-fatal.html'
            , 'htmlhead'            => '/data/www/lxr/templates/html/html-head-btn.html'
            , 'htmltail'            => '/data/www/lxr/templates/html/html-tail.html'
            , 'htmldir'                     => '/data/www/lxr/templates/html/html-dir-indexing.html'
            , 'htmlident'           => '/data/www/lxr/templates/html/html-ident.html'
            , 'htmlsearch'          => '/data/www/lxr/templates/html/html-search-glimpse.html'
            , 'htmlconfig'          => '/data/www/lxr/templates/html/html-config.html'
            , 'showconfighead'              => '/data/www/lxr/templates/html/config-head-btn-smaller.html'
            , 'diffleftwidth'       => 50
            , 'stylesheet'          => 'templates/lxr.css'
            , 'alternate_stylesheet' => [ 'templates/classic.css' ]
            , 'encoding'    => 'utf-8'
            , 'ignorefiles' =>
                    '^\\.|~$|\\.(o|a|orig)$|^CVS$|^core$'
            , 'graphicfile' =>
                    'bitmap|bmp|gif|icon?|jp2|jpe?g|pjpe?g|png|svg|tiff?|xbm|xpm'
            , 'filetypeconf' => '/data/www/lxr/templates/filetype.conf'
            , 'genericconf' => '/data/www/lxr/lib/LXR/Lang/generic.conf'
            , 'dbuser'              => 'lxr'
            , 'dbpass'              => 'xxxxx'
            , 'dbprefix'    => 'lxr_'
            }
    ,       {
              'virtroot'     => ''
            ,
              'caption'      => 'python'
            , 'shortcaption' => 'python'
            , 'treename'     => 'python'
            , 'sourceroot' => '/data/code/python'
            , 'sourcerootname' => '$v'
            , 'variables' =>
                    { 'v' =>
                            { 'name' => 'Version'
                            , 'range' => [qw(
                                            2.7
                                                    )]
                            }
                    }
            , 'dbname' => 'dbi:mysql:dbname=python'
            }
    )
  5. Run script create new mysql user for tree and init database:
    ./custom.d/initdb.sh
  6. Re-grant sql access for lxr user:
    GRANT ALL privileges ON python.* TO 'lxr'@'localhost' IDENTIFIED BY 'xxxxxx';
  7. Generate Index
     ./genxref --url=http://searchcode.babies.vn --tree=python --version=2.7

config apache for running first tree source

NameVirtualHost *:80
<VirtualHost *:80>
    DocumentRoot    /data/www/lxr
    ServerName              searchcode.babies.vn
</VirtualHost>
<IfDefine MODPERL2>
    PerlSwitches -T
</IfDefine>
<IfDefine !MODPERL2>
    PerlTaintCheck On
</IfDefine>

<IfModule mod_version.c>
    <IfDefine MODPERL2>
        PerlPostConfigRequire /data/www/lxr/custom.d/apache2-require.pl
    </IfDefine>
    <IfDefine !MODPERL2>
        PerlRequire /data/www/lxr/custom.d/apache2-require.pl
    </IfDefine>
</IfModule>

<Directory "/data/www/lxr">
    Options FollowSymLinks
    AllowOverride AuthConfig FileInfo Limit Options
    Order allow,deny
    Allow from all
</Directory>

Create and config new tree source with perl script

  • Run script create new tree source with below change configuration:
    ./scripts/configure-lxr.pl --add
    --- Caption in page header? (e.g. Project XYZZY displayed by LXR) > drupal
    --- Short title for button? (e.g. XYZZY) > drupal
    --- Tree identification in URL? (e.g. the-tree) > drupal
    --- Source directory? (e.g. /home/myself/project-tree) > /data/code/drupal
    --- Version name?  > 7.28
  • Update configuration:Copy the augmented lxr.conf configuration file and physically create the database:
    cp ./custom.d/lxr.conf .
    ./custom.d/initdb.sh
  • Index your trees: Run the genxref script for each tree
     ./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28

Create and config new tree source no perl script

  • grant for user lxr access database drupal:
    GRANT ALL privileges ON drupal.* TO 'lxr'@'localhost';
  • create perl script initdb_drupal.sh:
    echo "*** MySQL - Creating tree database drupal"
    mysql -u lxr -plxr12[email protected] <<END_OF_CREATE
    drop database if exists drupal;
    create database drupal;
    END_OF_CREATE
     
    echo "*** MySQL - Configuring tables lxr_ in database drupal"
    mysql -u lxr -plxr12[email protected] <<END_OF_TEMPLATE
    use drupal;
     
    drop table if exists lxr_filenum;
    drop table if exists lxr_symnum;
    drop table if exists lxr_typenum;
     
    create table lxr_filenum
    	( rcd int primary key
    	, fid int
    	);
    insert into lxr_filenum
    	(rcd, fid) VALUES (0, 0);
     
    create table lxr_symnum
    	( rcd int primary key
    	, sid int
    	);
    insert into lxr_symnum
    	(rcd, sid) VALUES (0, 0);
     
    create table lxr_typenum
    	( rcd int primary key
    	, tid int
    	);
    insert into lxr_typenum
    	(rcd, tid) VALUES (0, 0);
     
    alter table lxr_filenum
    	engine = MyISAM;
    alter table lxr_symnum
    	engine = MyISAM;
    alter table lxr_typenum
    	engine = MyISAM;
     
     
    /* Base version of files */
    /*	revision:	a VCS generated unique id for this version
    				of the file
     */
    create table lxr_files
    	( fileid    int                not null primary key
    	, filename  varbinary(255)     not null
    	, revision  varbinary(255)     not null
    	, constraint lxr_uk_files
    		unique (filename, revision)
    	, index lxr_filelookup (filename)
    	)
    	engine = MyISAM;
     
    /* Status of files in the DB */
    /*	fileid:		refers to base version
     *	relcount:	number of releases associated with base version
     *	indextime:	time when file was parsed for references
     *	status:		set of bits with the following meaning
     *		1	declaration have been parsed
     *		2	references have been processed
     *	Though this table could be merged with 'files',
     *	performance is improved with access to a very small item.
     */
    /* Deletion of a record automatically removes the associated
     * base version files record.
     */
    create table lxr_status
    	( fileid    int     not null primary key
    	, relcount  int
    	, indextime int
    	, status    tinyint not null
    	, constraint lxr_fk_sts_file
    		foreign key (fileid)
    		references lxr_files(fileid)
    	)
    	engine = MyISAM;
     
    /* The following trigger deletes no longer referenced files
     * (from releases), once status has been deleted so that
     * foreign key constrained has been cleared.
     */
    drop trigger if exists lxr_remove_file;
    create trigger lxr_remove_file
    	after delete on lxr_status
    	for each row
    		delete from lxr_files
    			where fileid = old.fileid;
     
    /* Aliases for files */
    /*	A base version may be known under several releaseids
     *	if it did not change in-between.
     *	fileid:		refers to base version
     *	releaseid:	"public" release tag
     */
    create table lxr_releases 
    	( fileid    int            not null
    	, releaseid varbinary(255) not null
    	, constraint lxr_pk_releases
    		primary key (fileid, releaseid)
    	, constraint lxr_fk_rls_fileid
    		foreign key (fileid)
    		references lxr_files(fileid)
    	)
    	engine = MyISAM;
     
    /* The following triggers maintain relcount integrity
     * in status table after insertion/deletion of releases
     */
    drop trigger if exists lxr_add_release;
    create trigger lxr_add_release
    	after insert on lxr_releases
    	for each row
    		update lxr_status
    			set relcount = relcount + 1
    			where fileid = new.fileid;
    /* Note: a release is erased only when option --reindexall
     * is given to genxref; it is thus necessary to reset status
     * to cause reindexing, especially if the file is shared by
     * several releases
     */
    drop trigger if exists lxr_remove_release;
    create trigger lxr_remove_release
    	after delete on lxr_releases
    	for each row
    		update lxr_status
    			set	relcount = relcount - 1
    -- 			,	status = 0
    			where fileid = old.fileid
    			and relcount > 0;
     
    /* Types for a language */
    /*	declaration:	provided by generic.conf
     */
    create table lxr_langtypes
    	( typeid       smallint         not null               
    	, langid       tinyint unsigned not null
    	, declaration  varchar(255)     not null
    	, constraint lxr_pk_langtypes
    		primary key  (typeid, langid)
    	)
    	engine = MyISAM;
     
    /* Symbol name dictionary */
    /*	symid:		unique symbol id for name
     * 	symcount:	number of definitions and usages for this name
     *	symname:	symbol name
     */
    create table lxr_symbols
    	( symid    int            not null                primary key
    	, symcount int
    	, symname  varbinary(255) not null unique
    	)
    	engine = MyISAM;
     
    /* The following function decrements the symbol reference count
     * (to be used in triggers).
     */
    delimiter //
    create procedure lxr_decsym(in whichsym int)
    begin
    	update lxr_symbols
    		set	symcount = symcount - 1
    		where symid = whichsym
    		and symcount > 0;
    end//
    delimiter ;
     
    /* Definitions */
    /*	symid:	refers to symbol name
     *  fileid and line define the location of the declaration
     *	langid:	language id
     *	typeid:	language type id
     *	relid:	optional id of the englobing declaration
     *			(refers to another symbol, not a definition)
     */
    create table lxr_definitions
    	( symid   int              not null
    	, fileid  int              not null
    	, line    int              not null
    	, typeid  smallint         not null
    	, langid  tinyint unsigned not null
    	, relid   int
    	, index lxr_i_definitions (symid)
    	, constraint lxr_fk_defn_symid
    		foreign key (symid)
    		references lxr_symbols(symid)
    	, constraint lxr_fk_defn_fileid
    		foreign key (fileid)
    		references lxr_files(fileid)
    	, constraint lxr_fk_defn_type
    		foreign key (typeid, langid)
    		references lxr_langtypes(typeid, langid)
    	, constraint lxr_fk_defn_relid
    		foreign key (relid)
    		references lxr_symbols(symid)
    	)
    	engine = MyISAM;
     
    /* The following trigger maintains correct symbol reference count
     * after definition deletion.
     */
    delimiter //
    drop trigger if exists lxr_remove_definition;
    create trigger lxr_remove_definition
    	after delete on lxr_definitions
    	for each row
    	begin
    		call lxr_decsym(old.symid);
    		if old.relid is not null
    		then call lxr_decsym(old.relid);
    		end if;
    	end//
    delimiter ;
     
    /* Usages */
    create table lxr_usages
    	( symid   int not null
    	, fileid  int not null
    	, line    int not null
    	, index lxr_i_usages (symid)
    	, constraint lxr_fk_use_symid
    		foreign key (symid)
    		references lxr_symbols(symid)
    	, constraint lxr_fk_use_fileid
    		foreign key (fileid)
    		references lxr_files(fileid)
    	)
    	engine = MyISAM;
     
    /* The following trigger maintains correct symbol reference count
     * after usage deletion.
     */
    drop trigger if exists lxr_remove_usage;
    create trigger lxr_remove_usage
    	after delete on lxr_usages
    	for each row
    	call lxr_decsym(old.symid);
     
    delimiter //
    create procedure lxr_PurgeAll ()
    begin
    	set @old_check = @@session.foreign_key_checks;
    	set session foreign_key_checks = OFF;
    	truncate table lxr_filenum;
    	truncate table lxr_symnum;
    	truncate table lxr_typenum;
    	insert into lxr_filenum
    		(rcd, fid) VALUES (0, 0);
    	insert into lxr_symnum
    		(rcd, sid) VALUES (0, 0);
    	insert into lxr_typenum
    		(rcd, tid) VALUES (0, 0);
    	truncate table lxr_definitions;
    	truncate table lxr_usages;
    	truncate table lxr_langtypes;
    	truncate table lxr_symbols;
    	truncate table lxr_releases;
    	truncate table lxr_status;
    	truncate table lxr_files;
    	set session foreign_key_checks = @old_check;
    end//
    delimiter ;
    END_OF_TEMPLATE
  • add config for drupal in lxr.conf:
          ,{
              'virtroot'     => ''
            , 'caption'      => 'drupal'
            , 'shortcaption' => 'drupal'
            , 'treename'     => 'drupal'
            , 'sourceroot' => '/data/code/drupal'
            , 'sourcerootname' => '$v'
            , 'variables' =>
                    { 'v' =>
                            { 'name' => 'Version'
                            , 'range' => [qw(
                                            7.28
                                                    )]
                            }
                    }
            , 'dbname' => 'dbi:mysql:dbname=drupal'
            }
    
  • run Index for drupal:
     ./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28

OpenGrok

Requirements for Install

Requirements:

Install requirements on CentOS:

yum install java-1.7.0-openjdk
yum install tomcat6

Install on CentOS

wget https://java.net/projects/opengrok/downloads/download/opengrok-0.12.1.tar.gz
tar xf opengrok-0.12.1.tar.gz
cp -R opengrok-0.12.1 /usr/local

Index and install web application opengrok

  1. Deploy the web application
    cd /usr/local/opengrok/bin
    ./OpenGrok deploy

    ⇒ web opengrok will be installed to /var/lib/tomcat6/webapps/source.war or /usr/share/tomcat6/webapps/source.war

  2. Remove unused source in /data/code and run Index to generate project config files:
    cd /data/code
    for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done
    for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -f $i; done done

    And run Index:

    cd /usr/local/opengrok-0.12.1/bin/
    ./OpenGrok index /data/code/

    ⇒ Generate /var/opengrok and output:

    00:03:41 SEVERE: Failed to send configuration to localhost:2424 (is web application server running with opengrok deployed?)
    java.net.ConnectException: Connection refused
    ..................

    Port 2424 was configured in webapp /var/lib/tomcat6/webapps/source.war/WEB-INF/web.xml

  3. Check search code after index:
    java -cp ./opengrok.jar org.opensolaris.opengrok.search.Search -R /var/opengrok/etc/configuration.xml -f crawler.py

    ⇒ Display search result for string “crawler.py”

  4. Restart tomcat6:
    /etc/init.d/tomcat6 restart

    ⇒ After restart, webapp OpenGrok will start and listen on Port 2424 for java application OpenGrok.jar connet and transfer data after run Index

Config Tomcat and check

Understand directory structuer and environment in tomcat6

Directory structure of tomcat6 on centos:/usr/share/tomcat6/

drwxr-xr-x 2 root root   4096 Jun 14 09:43 bin
lrwxrwxrwx 1 root tomcat   12 Jun 14 09:43 conf -> /etc/tomcat6
lrwxrwxrwx 1 root root     23 Jun 14 09:43 lib -> /usr/share/java/tomcat6
lrwxrwxrwx 1 root root     16 Jun 14 09:43 logs -> /var/log/tomcat6
lrwxrwxrwx 1 root root     23 Jun 14 09:43 temp -> /var/cache/tomcat6/temp
lrwxrwxrwx 1 root root     24 Jun 14 09:43 webapps -> /var/lib/tomcat6/webapps
lrwxrwxrwx 1 root root     23 Jun 14 09:43 work -> /var/cache/tomcat6/work

And environment variables for tomcat server:

ps -eaf | grep tomcat

⇒ output:

/usr/lib/jvm/jre/bin/java 
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory 
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory 
-classpath :/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar 
-Dcatalina.base=/usr/share/tomcat6 
-Dcatalina.home=/usr/share/tomcat6 
-Djava.endorsed.dirs= 
-Djava.io.tmpdir=/var/cache/tomcat6/temp 
-Djava.util.logging.config.file=/usr/share/tomcat6/conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap 
start

Check search code

Rebuild OpenGrok from source

Requirements

Requirements:

  • JDK 1.7 or higher
  • Apache Ant 1.8 or later

Prepare requirements on centos:

  • Install JDK 1.7:
    yum install java-1.7.0-openjdk
    yum install java-1.7.0-openjdk-devel
  • Install Ant 1.8
    wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.gz
    tar xf apache-ant-1.8.0-bin.tar.gz
    cp -R apache-ant-1.8.0 /usr/local

Rebuild OpenGrok

  • Prepare environment:
    export JAVA_HOME=/usr/lib/jvm/java-1.7.0
    export PATH=$PATH:/usr/local/apache-ant-1.8.0/bin/
  • Build:
    wget https://github.com/OpenGrok/OpenGrok/archive/0.12.1.tar.gz
    tar xf 0.12.1.tar.gz
    cd OpenGrok-0.12.1
    ant

    ⇒ output:

    ...................
    -copy-lib-without-netbeans:
         [copy] Copying 1 file to /root/source/searchcodeengine/OpenGrok-0.12.1/dist/lib
         [copy] Copying 8 files to /root/source/searchcodeengine/OpenGrok-0.12.1/dist
    
    -post-jar:
          [war] Building war: /root/source/searchcodeengine/OpenGrok-0.12.1/dist/source.war
         [echo] Generating man page..
    
    jar:
    
    BUILD SUCCESSFUL
    Total time: 16 seconds

Edit for recognizing drupal file

Go to directory OpenGrok-0.12.1/src/org/opensolaris/opengrok/analysis/php

Reindex source

  1. Update new source code to /data/code, for example python2.7:
    rm -rf /data/code/python2.7
    cp -R /usr/local/lib/python2.7/ /data/code/
  2. Remove code unused:
    cd /data/code
    for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done
    for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -rf $i; done done
  3. reindex the /data/code
    cd /usr/local/opengrok-0.12.1/bin/
    ./OpenGrok index /data/code/
searchcodeengine.txt · Last modified: 2017/01/12 09:12 (external edit)