====== Search Code Engine ======
===== Comparion Seach Code Engine Tools =====
Reference at https://github.com/OpenGrok/OpenGrok/wiki/Comparison-with-Similar-Tools
^Feature^LXR^ctags^cscope^ViewVC^GNU GLOBAL^OpenGrok^
|Full text Search| Y| | Y| | Y| Y|
|Definition Search| P| Y| Y| | Y| Y|
|Identifier Search| Y| | Y| | Y| Y|
|Path search| Y| | Y| | Y| Y|
|History Search| | | | | | Y|
|Caller/Callee Search| | | Y| | P| |
|RegExp Search| | | Y| | Y| P (only wildcards)|
|Shows matching lines| | Y| Y| | Y| Y|
|Hierarchical Search| | | | | Y| Y|
|query syntax like AND, OR, field:| | | | | | Y|
|Incremental update| | | P| | Y| Y|
|RSS Feed| | | | Y| | Y|
|Syntax highlighting-Xref| Y| | | P| Y| Y|
|Interface for SCM| | | | Y| Y| Y|
|open source| Y| Y| Y| Y| Y| Y|
|Usable URLs| Y| N/A| N/A| | Y| Y|
|Individual file download| | N/A| N/A| Y| N/A| Y|
|Changes at directory level| | N/A| N/A| P| N/A| Y|
|Multi language support| P| Y| P| N/A| Y| Y|
|User interface| WEB| CLI| CLI| WEB| WEB / CLI| WEB / CLI|
|Static HTML| | | | | Y| P (for xrefs)|
|Input completion| | | | | Y| P (with greasemonkey script)|
|Built-in parser| | Y| Y| | Y| P (lexical analysis)|
|Plug-in parser| | | | | Y| |
|Integration with (exuberant) ctags| | N/A| | | Y| Y|
|Integration with cscope| | | N/A| | Y| |
|Integration with emacs| | Y| Y| | Y| |
|Integration with vim| | Y| Y| | Y| P (through https://github.com/jdevera/vim-opengrok-search)|
|Integration with doxygen| | | | | Y| |
|Integration with Netbeans| | | | | | |
|Integration with less| | Y| | | Y| |
|Integration with bash| | | | | Y| |
|Integration with idutils| | | | | Y| |
|Integration with Elvis| | Y| | | Y|
And about language support, OpenGrok support the most programming languages:bzip2, C, C++, C#, ELF, Fortran, generic data, gzip, Java, Java archive, Java class, Java script, Lisp, Perl, PHP, plaintext, Python, Scala, Shell, SQL, PLSQL, tar, Tcl, troff, Visual basic, XML, zip
===== Ctags =====
Install Ctags:
  * Download Ctags at link http://ctags.sourceforge.net/
  * Install 
tar xf ctags-5.8.tar.gz
cd ctags-5.8
./configure
make
make install
===== lxr =====
==== Install and config lxr with Ctags ====
Get lxr at link http://sourceforge.net/projects/lxr/
=== Pre-requisites ===
Run below command to check packages missing:
./genxref --checkonly
  * Install perl File:MMagic
yum install perl-ExtUtils-MakeMaker
wget http://www.cpan.org/authors/id/K/KN/KNOK/File-MMagic-1.30.tar.gz
tar xf File-MMagic-1.30.tar.gz
cd File-MMagic-1.30
perl perl Makefile.PL
make & make install
  * Install perl Glimpse:
yum install perl-Module-Build
yum install perl-Test-Pod
yum install perl-Test-Pod-Coverage
wget http://webglimpse.net/trial/glimpse-latest.tar.gz
tar xf glimpse-latest.tar.gz
cd glimpse-4.18.6/
./configure
make & make install
  * Install mod_perl for apache:
yum install perl-ExtUtils-Embed
wget http://mirrors.digipower.vn/apache/perl/mod_perl-2.0.8.tar.gz
tar xf mod_perl-2.0.8.tar.gz
cd mod_perl-2.0.8
perl Makefile.PL MP_APXS=/usr/local/apache/bin/apxs
make & make install
add config mod_perl to /usr/local/apache/conf/httpd.conf:LoadModule perl_module modules/mod_perl.so
  * Install mod_version:
/usr/local/apache/bin/httpd -M | grep version
version_module (static)
Syntax OK
 => mode_version has been installed
=== Install and Config lxr ===
  - Install:
mkdir -p /data/www/lxr
cp -R lxr-2.0.2/* /data/www/lxr
  - Run script configure lxr with defauld configuration and below changes:
./scripts/configure-lxr.pl -vv
Configure for single/multiple trees? [S/m] > m
Server type? [dedicated/SHARED] > dedicated
--- Host name or IP? [//localhost] > http://searchcode.babies.vn
--- DB user name? [lxr] >
--- DB password ? [lxrpw] > xxxxx
--- Directory for glimpse databases? >/data/glimpsedirbase
Is your Apache version 2.4 or higher? [YES/no] > no
 And config for first tree:
--- Caption in page header? (e.g. Project XYZZY displayed by LXR) > python
--- Short title for button? (e.g. XYZZY) > python
--- Tree identification in URL? (e.g. the-tree) > python
--- Source directory? (e.g. /home/myself/project-tree) > /data/code
--- Database name? > python
--- Version name?  > 2.7
*** Configure another tree? [YES/no] > no
  - prepare config and check
cp custom.d/lxr.conf
./genxref --checkonly
Output:
[  OK  ]     Perl     version ... 5.10.1
[  OK  ]     ctags    version ... 5.8
Checked:    glimpse   version ... 4.18.5
Checked: glimpseindex version ... 4.18.5
Parameter 'swishbin' not defined - trying to find swish-e
swish-e not found, `command -v swish-e` returned a null string
genxref stopped without indexing by --checkonly option
  - Edit config lxr.conf:
'glimpsedirbase' => '/data/glimpsedirbase'
'sourceroot' => '/data/code'
 'range' => [qw(python2.7)]
 And below is new config:
(
        {
          'routing' => 'argument'
        , 'tmpdir' => '/tmp'
        , 'glimpsebin'     => '/usr/local/bin/glimpse'
        , 'glimpseindex'   => '/usr/local/bin/glimpseindex'
        , 'glimpsedirbase' => '/data/glimpsedirbase'
        , 'swishconf' => '/data/www/lxr/templates/swish-e.conf'
        , 'ectagsbin' => '/usr/local/bin/ctags'
        , 'ectagsconf' => '/data/www/lxr/templates/ectags.conf'
        , 'cvspath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
        , 'gitpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
        , 'hgpath'  => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
        , 'svnpath' => '/bin:/usr/local/bin:/usr/bin:/usr/sbin'
        ,       'host_names' => [ 'http://searchcode.babies.vn'
                                                ]
        , 'htmlfatal'           => '/data/www/lxr/templates/html/html-fatal.html'
        , 'htmlhead'            => '/data/www/lxr/templates/html/html-head-btn.html'
        , 'htmltail'            => '/data/www/lxr/templates/html/html-tail.html'
        , 'htmldir'                     => '/data/www/lxr/templates/html/html-dir-indexing.html'
        , 'htmlident'           => '/data/www/lxr/templates/html/html-ident.html'
        , 'htmlsearch'          => '/data/www/lxr/templates/html/html-search-glimpse.html'
        , 'htmlconfig'          => '/data/www/lxr/templates/html/html-config.html'
        , 'showconfighead'              => '/data/www/lxr/templates/html/config-head-btn-smaller.html'
        , 'diffleftwidth'       => 50
        , 'stylesheet'          => 'templates/lxr.css'
        , 'alternate_stylesheet' => [ 'templates/classic.css' ]
        , 'encoding'    => 'utf-8'
        , 'ignorefiles' =>
                '^\\.|~$|\\.(o|a|orig)$|^CVS$|^core$'
        , 'graphicfile' =>
                'bitmap|bmp|gif|icon?|jp2|jpe?g|pjpe?g|png|svg|tiff?|xbm|xpm'
        , 'filetypeconf' => '/data/www/lxr/templates/filetype.conf'
        , 'genericconf' => '/data/www/lxr/lib/LXR/Lang/generic.conf'
        , 'dbuser'              => 'lxr'
        , 'dbpass'              => 'xxxxx'
        , 'dbprefix'    => 'lxr_'
        }
,       {
          'virtroot'     => ''
        ,
          'caption'      => 'python'
        , 'shortcaption' => 'python'
        , 'treename'     => 'python'
        , 'sourceroot' => '/data/code/python'
        , 'sourcerootname' => '$v'
        , 'variables' =>
                { 'v' =>
                        { 'name' => 'Version'
                        , 'range' => [qw(
                                        2.7
                                                )]
                        }
                }
        , 'dbname' => 'dbi:mysql:dbname=python'
        }
)
  - Run script create new mysql user for tree and init database:
./custom.d/initdb.sh
  - Re-grant sql access for lxr user: 
grant all privileges on python.* to 'lxr'@'localhost' identified by 'xxxxxx';
  - Generate Index
 ./genxref --url=http://searchcode.babies.vn --tree=python --version=2.7
=== config apache for running first tree source ===
NameVirtualHost *:80
    DocumentRoot    /data/www/lxr
    ServerName              searchcode.babies.vn
    PerlSwitches -T
    PerlTaintCheck On
    
        PerlPostConfigRequire /data/www/lxr/custom.d/apache2-require.pl
    
    
        PerlRequire /data/www/lxr/custom.d/apache2-require.pl
    
    Options FollowSymLinks
    AllowOverride AuthConfig FileInfo Limit Options
    Order allow,deny
    Allow from all
==== Create and config new tree source with perl script ====
  * Run script create new tree source with below change configuration:
./scripts/configure-lxr.pl --add
--- Caption in page header? (e.g. Project XYZZY displayed by LXR) > drupal
--- Short title for button? (e.g. XYZZY) > drupal
--- Tree identification in URL? (e.g. the-tree) > drupal
--- Source directory? (e.g. /home/myself/project-tree) > /data/code/drupal
--- Version name?  > 7.28
  * Update configuration:Copy the augmented lxr.conf configuration file and physically create the database:
cp ./custom.d/lxr.conf .
./custom.d/initdb.sh
  * Index your trees: Run the genxref script for each tree
 ./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28
==== Create and config new tree source no perl script ====
  * grant for user lxr access database drupal:
GRANT ALL privileges ON drupal.* TO 'lxr'@'localhost';
  * create perl script initdb_drupal.sh:
echo "*** MySQL - Creating tree database drupal"
mysql -u lxr -plxr12!@ < 0;
/* Types for a language */
/*	declaration:	provided by generic.conf
 */
create table lxr_langtypes
	( typeid       smallint         not null               
	, langid       tinyint unsigned not null
	, declaration  varchar(255)     not null
	, constraint lxr_pk_langtypes
		primary key  (typeid, langid)
	)
	engine = MyISAM;
/* Symbol name dictionary */
/*	symid:		unique symbol id for name
 * 	symcount:	number of definitions and usages for this name
 *	symname:	symbol name
 */
create table lxr_symbols
	( symid    int            not null                primary key
	, symcount int
	, symname  varbinary(255) not null unique
	)
	engine = MyISAM;
/* The following function decrements the symbol reference count
 * (to be used in triggers).
 */
delimiter //
create procedure lxr_decsym(in whichsym int)
begin
	update lxr_symbols
		set	symcount = symcount - 1
		where symid = whichsym
		and symcount > 0;
end//
delimiter ;
/* Definitions */
/*	symid:	refers to symbol name
 *  fileid and line define the location of the declaration
 *	langid:	language id
 *	typeid:	language type id
 *	relid:	optional id of the englobing declaration
 *			(refers to another symbol, not a definition)
 */
create table lxr_definitions
	( symid   int              not null
	, fileid  int              not null
	, line    int              not null
	, typeid  smallint         not null
	, langid  tinyint unsigned not null
	, relid   int
	, index lxr_i_definitions (symid)
	, constraint lxr_fk_defn_symid
		foreign key (symid)
		references lxr_symbols(symid)
	, constraint lxr_fk_defn_fileid
		foreign key (fileid)
		references lxr_files(fileid)
	, constraint lxr_fk_defn_type
		foreign key (typeid, langid)
		references lxr_langtypes(typeid, langid)
	, constraint lxr_fk_defn_relid
		foreign key (relid)
		references lxr_symbols(symid)
	)
	engine = MyISAM;
/* The following trigger maintains correct symbol reference count
 * after definition deletion.
 */
delimiter //
drop trigger if exists lxr_remove_definition;
create trigger lxr_remove_definition
	after delete on lxr_definitions
	for each row
	begin
		call lxr_decsym(old.symid);
		if old.relid is not null
		then call lxr_decsym(old.relid);
		end if;
	end//
delimiter ;
/* Usages */
create table lxr_usages
	( symid   int not null
	, fileid  int not null
	, line    int not null
	, index lxr_i_usages (symid)
	, constraint lxr_fk_use_symid
		foreign key (symid)
		references lxr_symbols(symid)
	, constraint lxr_fk_use_fileid
		foreign key (fileid)
		references lxr_files(fileid)
	)
	engine = MyISAM;
/* The following trigger maintains correct symbol reference count
 * after usage deletion.
 */
drop trigger if exists lxr_remove_usage;
create trigger lxr_remove_usage
	after delete on lxr_usages
	for each row
	call lxr_decsym(old.symid);
delimiter //
create procedure lxr_PurgeAll ()
begin
	set @old_check = @@session.foreign_key_checks;
	set session foreign_key_checks = OFF;
	truncate table lxr_filenum;
	truncate table lxr_symnum;
	truncate table lxr_typenum;
	insert into lxr_filenum
		(rcd, fid) VALUES (0, 0);
	insert into lxr_symnum
		(rcd, sid) VALUES (0, 0);
	insert into lxr_typenum
		(rcd, tid) VALUES (0, 0);
	truncate table lxr_definitions;
	truncate table lxr_usages;
	truncate table lxr_langtypes;
	truncate table lxr_symbols;
	truncate table lxr_releases;
	truncate table lxr_status;
	truncate table lxr_files;
	set session foreign_key_checks = @old_check;
end//
delimiter ;
END_OF_TEMPLATE
  * add config for drupal in lxr.conf:
      ,{
          'virtroot'     => ''
        , 'caption'      => 'drupal'
        , 'shortcaption' => 'drupal'
        , 'treename'     => 'drupal'
        , 'sourceroot' => '/data/code/drupal'
        , 'sourcerootname' => '$v'
        , 'variables' =>
                { 'v' =>
                        { 'name' => 'Version'
                        , 'range' => [qw(
                                        7.28
                                                )]
                        }
                }
        , 'dbname' => 'dbi:mysql:dbname=drupal'
        }
  * run Index for drupal:
 ./genxref --url=http://searchcode.babies.vn --tree=drupal --version=7.28
===== OpenGrok =====
{{:other:setup-project.png|}}
==== Requirements for Install ====
Requirements:
  * [[http://www.oracle.com/technetwork/java/|JDK 1.7]] or higher
  * [[searchcodeengine#ctags|Exuberant Ctags]] for analysis
  * Webserver [[http://tomcat.apache.org/|Tomcat]]
Install requirements on CentOS:
yum install java-1.7.0-openjdk
yum install tomcat6
==== Install on CentOS ====
wget https://java.net/projects/opengrok/downloads/download/opengrok-0.12.1.tar.gz
tar xf opengrok-0.12.1.tar.gz
cp -R opengrok-0.12.1 /usr/local
==== Index and install web application opengrok ====
  - Deploy the web application
cd /usr/local/opengrok/bin
./OpenGrok deploy
 => web opengrok will be installed to **/var/lib/tomcat6/webapps/source.war** or **/usr/share/tomcat6/webapps/source.war**
  - Remove unused source in /data/code and run Index to generate project config files:
cd /data/code
for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done
for ext in pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -f $i; done done
 And run Index:
cd /usr/local/opengrok-0.12.1/bin/
./OpenGrok index /data/code/
 => Generate **/var/opengrok** and output:
00:03:41 SEVERE: Failed to send configuration to localhost:2424 (is web application server running with opengrok deployed?)
java.net.ConnectException: Connection refused
..................
Port 2424 was configured in webapp **/var/lib/tomcat6/webapps/source.war/WEB-INF/web.xml**
  - Check search code after index:
java -cp ./opengrok.jar org.opensolaris.opengrok.search.Search -R /var/opengrok/etc/configuration.xml -f crawler.py
 => Display search result for string "crawler.py"
  - Restart tomcat6:
/etc/init.d/tomcat6 restart
 => After restart, webapp OpenGrok will start and **listen on Port 2424** for java application OpenGrok.jar connet and transfer data after run Index
==== Config Tomcat and check ====
=== Understand directory structuer and environment in tomcat6 ===
Directory structure of tomcat6 on centos:**/usr/share/tomcat6/**
drwxr-xr-x 2 root root   4096 Jun 14 09:43 bin
lrwxrwxrwx 1 root tomcat   12 Jun 14 09:43 conf -> /etc/tomcat6
lrwxrwxrwx 1 root root     23 Jun 14 09:43 lib -> /usr/share/java/tomcat6
lrwxrwxrwx 1 root root     16 Jun 14 09:43 logs -> /var/log/tomcat6
lrwxrwxrwx 1 root root     23 Jun 14 09:43 temp -> /var/cache/tomcat6/temp
lrwxrwxrwx 1 root root     24 Jun 14 09:43 webapps -> /var/lib/tomcat6/webapps
lrwxrwxrwx 1 root root     23 Jun 14 09:43 work -> /var/cache/tomcat6/work
And environment variables for tomcat server:
ps -eaf | grep tomcat
 => output:
/usr/lib/jvm/jre/bin/java 
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory 
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory 
-classpath :/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar 
-Dcatalina.base=/usr/share/tomcat6 
-Dcatalina.home=/usr/share/tomcat6 
-Djava.endorsed.dirs= 
-Djava.io.tmpdir=/var/cache/tomcat6/temp 
-Djava.util.logging.config.file=/usr/share/tomcat6/conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap 
start
=== Check search code ===
goto link:http://128.199.236.122:8080/source/
==== Rebuild OpenGrok from source ====
=== Requirements ===
Requirements:
  * JDK 1.7 or higher
  * Apache Ant 1.8 or later
Prepare requirements on centos:
  * Install JDK 1.7:
yum install java-1.7.0-openjdk
yum install java-1.7.0-openjdk-devel
  * Install Ant 1.8 
wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.gz
tar xf apache-ant-1.8.0-bin.tar.gz
cp -R apache-ant-1.8.0 /usr/local
=== Rebuild OpenGrok ===
  * Prepare environment:
export JAVA_HOME=/usr/lib/jvm/java-1.7.0
export PATH=$PATH:/usr/local/apache-ant-1.8.0/bin/
  * Build:
wget https://github.com/OpenGrok/OpenGrok/archive/0.12.1.tar.gz
tar xf 0.12.1.tar.gz
cd OpenGrok-0.12.1
ant
 => output:
...................
-copy-lib-without-netbeans:
     [copy] Copying 1 file to /root/source/searchcodeengine/OpenGrok-0.12.1/dist/lib
     [copy] Copying 8 files to /root/source/searchcodeengine/OpenGrok-0.12.1/dist
-post-jar:
      [war] Building war: /root/source/searchcodeengine/OpenGrok-0.12.1/dist/source.war
     [echo] Generating man page..
jar:
BUILD SUCCESSFUL
Total time: 16 seconds
=== Edit for recognizing drupal file ===
Go to directory **OpenGrok-0.12.1/src/org/opensolaris/opengrok/analysis/php**
==== Reindex source ====
  - Update new source code to /data/code, for example python2.7:
rm -rf /data/code/python2.7
cp -R /usr/local/lib/python2.7/ /data/code/
  - Remove code unused:
cd /data/code
for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i; done done
for ext in dll gif ico pxd o git hg svn pyo pyc so gz zip; do echo "*.$ext";for i in `find . -name "*.$ext"`; do echo $i;rm -rf $i; done done
  - reindex the /data/code
cd /usr/local/opengrok-0.12.1/bin/
./OpenGrok index /data/code/