Search Code Engine

Reference at https://github.com/OpenGrok/OpenGrok/wiki/Comparison-with-Similar-Tools

And about language support, OpenGrok support the most programming languages:bzip2, C, C++, C#, ELF, Fortran, generic data, gzip, Java, Java archive, Java class, Java script, Lisp, Perl, PHP, plaintext, Python, Scala, Shell, SQL, PLSQL, tar, Tcl, troff, Visual basic, XML, zip

Install Ctags:

• Download Ctags at link http://ctags.sourceforge.net/
• Install

  tar xf ctags-5.8.tar.gz
  cd ctags-5.8
  ./configure
  make
  make install

Get lxr at link http://sourceforge.net/projects/lxr/

Run below command to check packages missing:

./genxref --checkonly

• Install perl File:MMagic

  yum install perl-ExtUtils-MakeMaker
  wget http://www.cpan.org/authors/id/K/KN/KNOK/File-MMagic-1.30.tar.gz
  tar xf File-MMagic-1.30.tar.gz
  cd File-MMagic-1.30
  perl perl Makefile.PL
  make & make install

• Install perl Glimpse:

  yum install perl-Module-Build
  yum install perl-Test-Pod
  yum install perl-Test-Pod-Coverage
  wget http://webglimpse.net/trial/glimpse-latest.tar.gz
  tar xf glimpse-latest.tar.gz
  cd glimpse-4.18.6/
  ./configure
  make & make install

• Install mod_perl for apache:

  yum install perl-ExtUtils-Embed
  wget http://mirrors.digipower.vn/apache/perl/mod_perl-2.0.8.tar.gz
  tar xf mod_perl-2.0.8.tar.gz
  cd mod_perl-2.0.8
  perl Makefile.PL MP_APXS=/usr/local/apache/bin/apxs
  make & make install

  add config mod_perl to /usr/local/apache/conf/httpd.conf:

  LoadModule perl_module modules/mod_perl.so

• Install mod_version:

  /usr/local/apache/bin/httpd -M | grep version
  version_module (static)
  Syntax OK

  ⇒ mode_version has been installed

1. Install:

   mkdir -p /data/www/lxr
   cp -R lxr-2.0.2/* /data/www/lxr

2. Run script configure lxr with defauld configuration and below changes:

   ./scripts/configure-lxr.pl -vv
   Configure for single/multiple trees? [S/m] > m
   Server type? [dedicated/SHARED] > dedicated
   --- Host name or IP? [//localhost] > http://searchcode.babies.vn
   --- DB user name? [lxr] >
   --- DB password ? [lxrpw] > xxxxx
   --- Directory for glimpse databases? >/data/glimpsedirbase
   Is your Apache version 2.4 or higher? [YES/no] > no

   And config for first tree:

   --- Caption in page header? (e.g. Project XYZZY displayed by LXR) > python
   --- Short title for button? (e.g. XYZZY) > python
   --- Tree identification in URL? (e.g. the-tree) > python
   --- Source directory? (e.g. /home/myself/project-tree) > /data/code
   --- Database name? > python
   --- Version name?  > 2.7
   *** Configure another tree? [YES/no] > no

3. prepare config and check

   cp custom.d/lxr.conf
   ./genxref --checkonly

   Output:

   [  OK  ]     Perl     version ... 5.10.1
   [  OK  ]     ctags    version ... 5.8
   Checked:    glimpse   version ... 4.18.5
   Checked: glimpseindex version ... 4.18.5
   Parameter 'swishbin' not defined - trying to find swish-e
   swish-e not found, `command -v swish-e` returned a null string
   genxref stopped without indexing by --checkonly option

4. Edit config lxr.conf:

   'glimpsedirbase' => '/data/glimpsedirbase'
   'sourceroot' => '/data/code'
    'range' => [qw(python2.7)]

   And below is new config:

   (
           {
             'routing' => 'argument'
           , 'tmpdir' => '/tmp'
           , 'glimpsebin'     => '/usr/local/bin/glimpse'
           , 'glimpseindex'   => '/usr/local/bin/glimpseindex'
           , 'glimpsedirbase' => '/data/glimpsedirbase'
           , 'swishconf' => '/data/www/lxr/templates/swish-e.conf'
           , 'ectagsbin' => '/usr/local/bin/ctags'
           , 'ectagsconf' => '/data/www/lxr/templates/ectags.conf'
           , 'cvspath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
           , 'gitpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
           , 'hgpath'  => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
           , 'svnpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
           ,       'host_names' => [ 'http://searchcode.babies.vn'
                                                   ]
           , 'htmlfatal'           => '/data/www/lxr/templates/html/html-fatal.html'
           , 'htmlhead'            => '/data/www/lxr/templates/html/html-head-btn.html'
           , 'htmltail'            => '/data/www/lxr/templates/html/html-tail.html'
           , 'htmldir'                     => '/data/www/lxr/templates/html/html-dir-indexing.html'
           , 'htmlident'           => '/data/www/lxr/templates/html/html-ident.html'
           , 'htmlsearch'          => '/data/www/lxr/templates/html/html-search-glimpse.html'
           , 'htmlconfig'          => '/data/www/lxr/templates/html/html-config.html'
           , 'showconfighead'              => '/data/www/lxr/templates/html/config-head-btn-smaller.html'
           , 'diffleftwidth'       => 50
           , 'stylesheet'          => 'templates/lxr.css'
           , 'alternate_stylesheet' => [ 'templates/classic.css' ]
           , 'encoding'    => 'utf-8'
           , 'ignorefiles' =>
                   '^\\.|~$|\\.(o|a|orig)$|^CVS$|^core$'
           , 'graphicfile' =>
                   'bitmap|bmp|gif|icon?|jp2|jpe?g|pjpe?g|png|svg|tiff?|xbm|xpm'
           , 'filetypeconf' => '/data/www/lxr/templates/filetype.conf'
           , 'genericconf' => '/data/www/lxr/lib/LXR/Lang/generic.conf'
           , 'dbuser'              => 'lxr'
           , 'dbpass'              => 'xxxxx'
           , 'dbprefix'    => 'lxr_'
           }
   ,       {
             'virtroot'     => ''
           ,
             'caption'      => 'python'
           , 'shortcaption' => 'python'
           , 'treename'     => 'python'
           , 'sourceroot' => '/data/code/python'
           , 'sourcerootname' => '$v'
           , 'variables' =>
                   { 'v' =>
                           { 'name' => 'Version'
                           , 'range' => [qw(
                                           2.7
                                                   )]
                           }
                   }
           , 'dbname' => 'dbi:mysql:dbname=python'
           }
   )

5. Run script create new mysql user for tree and init database:

   ./custom.d/initdb.sh

6. Re-grant sql access for lxr user:

   GRANT ALL privileges ON python.* TO 'lxr'@'localhost' IDENTIFIED BY 'xxxxxx';

7. Generate Index

    ./genxref --url=http://searchcode.babies.vn --tree=python --version=2.7

NameVirtualHost *:80
<VirtualHost *:80>
    DocumentRoot    /data/www/lxr
    ServerName              searchcode.babies.vn
</VirtualHost>
<IfDefine MODPERL2>
    PerlSwitches -T
</IfDefine>
<IfDefine !MODPERL2>
    PerlTaintCheck On
</IfDefine>

<IfModule mod_version.c>
    <IfDefine MODPERL2>
        PerlPostConfigRequire /data/www/lxr/custom.d/apache2-require.pl
    </IfDefine>
    <IfDefine !MODPERL2>
        PerlRequire /data/www/lxr/custom.d/apache2-require.pl
    </IfDefine>
</IfModule>

<Directory "/data/www/lxr">
    Options FollowSymLinks
    AllowOverride AuthConfig FileInfo Limit Options
    Order allow,deny
    Allow from all
</Directory>

• Run script create new tree source with below change configuration:

  ./scripts/configure-lxr.pl --add
  --- Caption in page header? (e.g. Project XYZZY displayed by LXR) > drupal
  --- Short title for button? (e.g. XYZZY) > drupal
  --- Tree identification in URL? (e.g. the-tree) > drupal
  --- Source directory? (e.g. /home/myself/project-tree) > /data/code/drupal
  --- Version name?  > 7.28

• Update configuration:Copy the augmented lxr.conf configuration file and physically create the database:

  cp ./custom.d/lxr.conf .
  ./custom.d/initdb.sh

• Index your trees: Run the genxref script for each tree

   ./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28

• grant for user lxr access database drupal:

  GRANT ALL privileges ON drupal.* TO 'lxr'@'localhost';

• create perl script initdb_drupal.sh:

  echo "*** MySQL - Creating tree database drupal"
  mysql -u lxr -plxr12!@ <<END_OF_CREATE
  drop database if exists drupal;
  create database drupal;
  END_OF_CREATE
   
  echo "*** MySQL - Configuring tables lxr_ in database drupal"
  mysql -u lxr -plxr12!@ <<END_OF_TEMPLATE
  use drupal;
   
  drop table if exists lxr_filenum;
  drop table if exists lxr_symnum;
  drop table if exists lxr_typenum;
   
  create table lxr_filenum
  	( rcd int primary key
  	, fid int
  	);
  insert into lxr_filenum
  	(rcd, fid) VALUES (0, 0);
   
  create table lxr_symnum
  	( rcd int primary key
  	, sid int
  	);
  insert into lxr_symnum
  	(rcd, sid) VALUES (0, 0);
   
  create table lxr_typenum
  	( rcd int primary key
  	, tid int
  	);
  insert into lxr_typenum
  	(rcd, tid) VALUES (0, 0);
   
  alter table lxr_filenum
  	engine = MyISAM;
  alter table lxr_symnum
  	engine = MyISAM;
  alter table lxr_typenum
  	engine = MyISAM;
   
   
  /* Base version of files */
  /*	revision:	a VCS generated unique id for this version
  				of the file
   */
  create table lxr_files
  	( fileid    int                not null primary key
  	, filename  varbinary(255)     not null
  	, revision  varbinary(255)     not null
  	, constraint lxr_uk_files
  		unique (filename, revision)
  	, index lxr_filelookup (filename)
  	)
  	engine = MyISAM;
   
  /* Status of files in the DB */
  /*	fileid:		refers to base version
   *	relcount:	number of releases associated with base version
   *	indextime:	time when file was parsed for references
   *	status:		set of bits with the following meaning
   *		1	declaration have been parsed
   *		2	references have been processed
   *	Though this table could be merged with 'files',
   *	performance is improved with access to a very small item.
   */
  /* Deletion of a record automatically removes the associated
   * base version files record.
   */
  create table lxr_status
  	( fileid    int     not null primary key
  	, relcount  int
  	, indextime int
  	, status    tinyint not null
  	, constraint lxr_fk_sts_file
  		foreign key (fileid)
  		references lxr_files(fileid)
  	)
  	engine = MyISAM;
   
  /* The following trigger deletes no longer referenced files
   * (from releases), once status has been deleted so that
   * foreign key constrained has been cleared.
   */
  drop trigger if exists lxr_remove_file;
  create trigger lxr_remove_file
  	after delete on lxr_status
  	for each row
  		delete from lxr_files
  			where fileid = old.fileid;
   
  /* Aliases for files */
  /*	A base version may be known under several releaseids
   *	if it did not change in-between.
   *	fileid:		refers to base version
   *	releaseid:	"public" release tag
   */
  create table lxr_releases 
  	( fileid    int            not null
  	, releaseid varbinary(255) not null
  	, constraint lxr_pk_releases
  		primary key (fileid, releaseid)
  	, constraint lxr_fk_rls_fileid
  		foreign key (fileid)
  		references lxr_files(fileid)
  	)
  	engine = MyISAM;
   
  /* The following triggers maintain relcount integrity
   * in status table after insertion/deletion of releases
   */
  drop trigger if exists lxr_add_release;
  create trigger lxr_add_release
  	after insert on lxr_releases
  	for each row
  		update lxr_status
  			set relcount = relcount + 1
  			where fileid = new.fileid;
  /* Note: a release is erased only when option --reindexall
   * is given to genxref; it is thus necessary to reset status
   * to cause reindexing, especially if the file is shared by
   * several releases
   */
  drop trigger if exists lxr_remove_release;
  create trigger lxr_remove_release
  	after delete on lxr_releases
  	for each row
  		update lxr_status
  			set	relcount = relcount - 1
  -- 			,	status = 0
  			where fileid = old.fileid
  			and relcount > 0;
   
  /* Types for a language */
  /*	declaration:	provided by generic.conf
   */
  create table lxr_langtypes
  	( typeid       smallint         not null               
  	, langid       tinyint unsigned not null
  	, declaration  varchar(255)     not null
  	, constraint lxr_pk_langtypes
  		primary key  (typeid, langid)
  	)
  	engine = MyISAM;
   
  /* Symbol name dictionary */
  /*	symid:		unique symbol id for name
   * 	symcount:	number of definitions and usages for this name
   *	symname:	symbol name
   */
  create table lxr_symbols
  	( symid    int            not null                primary key
  	, symcount int
  	, symname  varbinary(255) not null unique
  	)
  	engine = MyISAM;
   
  /* The following function decrements the symbol reference count
   * (to be used in triggers).
   */
  delimiter //
  create procedure lxr_decsym(in whichsym int)
  begin
  	update lxr_symbols
  		set	symcount = symcount - 1
  		where symid = whichsym
  		and symcount > 0;
  end//
  delimiter ;
   
  /* Definitions */
  /*	symid:	refers to symbol name
   *  fileid and line define the location of the declaration
   *	langid:	language id
   *	typeid:	language type id
   *	relid:	optional id of the englobing declaration
   *			(refers to another symbol, not a definition)
   */
  create table lxr_definitions
  	( symid   int              not null
  	, fileid  int              not null
  	, line    int              not null
  	, typeid  smallint         not null
  	, langid  tinyint unsigned not null
  	, relid   int
  	, index lxr_i_definitions (symid)
  	, constraint lxr_fk_defn_symid
  		foreign key (symid)
  		references lxr_symbols(symid)
  	, constraint lxr_fk_defn_fileid
  		foreign key (fileid)
  		references lxr_files(fileid)
  	, constraint lxr_fk_defn_type
  		foreign key (typeid, langid)
  		references lxr_langtypes(typeid, langid)
  	, constraint lxr_fk_defn_relid
  		foreign key (relid)
  		references lxr_symbols(symid)
  	)
  	engine = MyISAM;
   
  /* The following trigger maintains correct symbol reference count
   * after definition deletion.
   */
  delimiter //
  drop trigger if exists lxr_remove_definition;
  create trigger lxr_remove_definition
  	after delete on lxr_definitions
  	for each row
  	begin
  		call lxr_decsym(old.symid);
  		if old.relid is not null
  		then call lxr_decsym(old.relid);
  		end if;
  	end//
  delimiter ;
   
  /* Usages */
  create table lxr_usages
  	( symid   int not null
  	, fileid  int not null
  	, line    int not null
  	, index lxr_i_usages (symid)
  	, constraint lxr_fk_use_symid
  		foreign key (symid)
  		references lxr_symbols(symid)
  	, constraint lxr_fk_use_fileid
  		foreign key (fileid)
  		references lxr_files(fileid)
  	)
  	engine = MyISAM;
   
  /* The following trigger maintains correct symbol reference count
   * after usage deletion.
   */
  drop trigger if exists lxr_remove_usage;
  create trigger lxr_remove_usage
  	after delete on lxr_usages
  	for each row
  	call lxr_decsym(old.symid);
   
  delimiter //
  create procedure lxr_PurgeAll ()
  begin
  	set @old_check = @@session.foreign_key_checks;
  	set session foreign_key_checks = OFF;
  	truncate table lxr_filenum;
  	truncate table lxr_symnum;
  	truncate table lxr_typenum;
  	insert into lxr_filenum
  		(rcd, fid) VALUES (0, 0);
  	insert into lxr_symnum
  		(rcd, sid) VALUES (0, 0);
  	insert into lxr_typenum
  		(rcd, tid) VALUES (0, 0);
  	truncate table lxr_definitions;
  	truncate table lxr_usages;
  	truncate table lxr_langtypes;
  	truncate table lxr_symbols;
  	truncate table lxr_releases;
  	truncate table lxr_status;
  	truncate table lxr_files;
  	set session foreign_key_checks = @old_check;
  end//
  delimiter ;
  END_OF_TEMPLATE

• add config for drupal in lxr.conf:

        ,{
            'virtroot'     => ''
          , 'caption'      => 'drupal'
          , 'shortcaption' => 'drupal'
          , 'treename'     => 'drupal'
          , 'sourceroot' => '/data/code/drupal'
          , 'sourcerootname' => '$v'
          , 'variables' =>
                  { 'v' =>
                          { 'name' => 'Version'
                          , 'range' => [qw(
                                          7.28
                                                  )]
                          }
                  }
          , 'dbname' => 'dbi:mysql:dbname=drupal'
          }
  

• run Index for drupal:

   ./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28

Requirements:

• JDK 1.7 or higher
• Exuberant Ctags for analysis
• Webserver Tomcat

Install requirements on CentOS:

yum install java-1.7.0-openjdk
yum install tomcat6

wget https://java.net/projects/opengrok/downloads/download/opengrok-0.12.1.tar.gz
tar xf opengrok-0.12.1.tar.gz
cp -R opengrok-0.12.1 /usr/local

1. Deploy the web application

   cd /usr/local/opengrok/bin
   ./OpenGrok deploy

   ⇒ web opengrok will be installed to /var/lib/tomcat6/webapps/source.war or /usr/share/tomcat6/webapps/source.war

2. Remove unused source in /data/code and run Index to generate project config files:

   cd /data/code
   for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done
   for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -f $i; done done

   And run Index:

   cd /usr/local/opengrok-0.12.1/bin/
   ./OpenGrok index /data/code/

   ⇒ Generate /var/opengrok and output:

   00:03:41 SEVERE: Failed to send configuration to localhost:2424 (is web application server running with opengrok deployed?)
   java.net.ConnectException: Connection refused
   ..................

   Port 2424 was configured in webapp /var/lib/tomcat6/webapps/source.war/WEB-INF/web.xml

3. Check search code after index:

   java -cp ./opengrok.jar org.opensolaris.opengrok.search.Search -R /var/opengrok/etc/configuration.xml -f crawler.py

   ⇒ Display search result for string “crawler.py”

4. Restart tomcat6:

   /etc/init.d/tomcat6 restart

   ⇒ After restart, webapp OpenGrok will start and listen on Port 2424 for java application OpenGrok.jar connet and transfer data after run Index

Directory structure of tomcat6 on centos:/usr/share/tomcat6/

drwxr-xr-x 2 root root   4096 Jun 14 09:43 bin
lrwxrwxrwx 1 root tomcat   12 Jun 14 09:43 conf -> /etc/tomcat6
lrwxrwxrwx 1 root root     23 Jun 14 09:43 lib -> /usr/share/java/tomcat6
lrwxrwxrwx 1 root root     16 Jun 14 09:43 logs -> /var/log/tomcat6
lrwxrwxrwx 1 root root     23 Jun 14 09:43 temp -> /var/cache/tomcat6/temp
lrwxrwxrwx 1 root root     24 Jun 14 09:43 webapps -> /var/lib/tomcat6/webapps
lrwxrwxrwx 1 root root     23 Jun 14 09:43 work -> /var/cache/tomcat6/work

And environment variables for tomcat server:

ps -eaf | grep tomcat

⇒ output:

/usr/lib/jvm/jre/bin/java 
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory 
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory 
-classpath :/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar 
-Dcatalina.base=/usr/share/tomcat6 
-Dcatalina.home=/usr/share/tomcat6 
-Djava.endorsed.dirs= 
-Djava.io.tmpdir=/var/cache/tomcat6/temp 
-Djava.util.logging.config.file=/usr/share/tomcat6/conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap 
start

goto link:http://128.199.236.122:8080/source/

Requirements:

• JDK 1.7 or higher
• Apache Ant 1.8 or later

Prepare requirements on centos:

• Install JDK 1.7:

  yum install java-1.7.0-openjdk
  yum install java-1.7.0-openjdk-devel

• Install Ant 1.8

  wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.gz
  tar xf apache-ant-1.8.0-bin.tar.gz
  cp -R apache-ant-1.8.0 /usr/local

• Prepare environment:

  export JAVA_HOME=/usr/lib/jvm/java-1.7.0
  export PATH=$PATH:/usr/local/apache-ant-1.8.0/bin/

• Build:

  wget https://github.com/OpenGrok/OpenGrok/archive/0.12.1.tar.gz
  tar xf 0.12.1.tar.gz
  cd OpenGrok-0.12.1
  ant

  ⇒ output:

  ...................
  -copy-lib-without-netbeans:
       [copy] Copying 1 file to /root/source/searchcodeengine/OpenGrok-0.12.1/dist/lib
       [copy] Copying 8 files to /root/source/searchcodeengine/OpenGrok-0.12.1/dist
  
  -post-jar:
        [war] Building war: /root/source/searchcodeengine/OpenGrok-0.12.1/dist/source.war
       [echo] Generating man page..
  
  jar:
  
  BUILD SUCCESSFUL
  Total time: 16 seconds

Go to directory OpenGrok-0.12.1/src/org/opensolaris/opengrok/analysis/php

1. Update new source code to /data/code, for example python2.7:

   rm -rf /data/code/python2.7
   cp -R /usr/local/lib/python2.7/ /data/code/

2. Remove code unused:

   cd /data/code
   for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done
   for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -rf $i; done done

3. reindex the /data/code

   cd /usr/local/opengrok-0.12.1/bin/
   ./OpenGrok index /data/code/