NEW IN ORCA 0.27b1 ================== 1) By default, now create an hourly plot that shows the last 1.5 hours of data. Add a global "generate_hourly_plot" configuration file parameter that when set to 0 turns this plot off. Hourly plot creation is disabled in orcallator.cfg.in since orcallator.se by default measures the system every 5 minutes and the plots look blocky. 2) Fix the warning message in src/orca.pl.in Use on uninitialized value in array element at (eval X) line 1, line Y when Orca was run with Perl 5.6.0. This bug was in the filename sorting subroutine. This problem is probably responsible for problems where there are missing data from generated plots. Since the filename sorting subroutine tells Orca the order in which to load data into the RRD files and the sorted filename list will have newer data files listed before older data files, once newer data is entered into an RRD file you cannot add older data, resulting in missing data in the output plots. If this is a problem, then the solution is to remove the RRD files and rerun Orca with all of the input data files. 3) Add a new command line option named -no-images which suppresses image generation. Remove the command line option -r which told Orca to only update the RRD files and not generate HTML and image files. To replace -r functionality, use both -no-images and -no-html command line options. 4) Add a new command line option -no-html to instruct Orca to skip generating HTML files. Patch from Alex Howansky . 5) Add a new command line option -logfile that specifies a filename that STDOUT and STDERR are redirected to so that all messages, warnings and errors are printed to the file. Now when a SIGPIPE is caught, messages will continue to be printed unless -logfile was not given to Orca. Patch supplied by Bruce Johnson . 6) Add a new command line option -daemon that puts Orca in the background or daemonizes it. It is recommended that when this command line option is used that -logfile is used. Patch supplied by Bruce Johnson . 7) A new contrib directory has been created which is where contributed scripts and programs will be kept. The first submissions are: a) Include the orcaservices package in the contrib directory which was written by Carlos Canau and documented by Jose Carlos . This package allows monitoring of many different types of Unix services, such as DNS, SMTP, POP, etc. b) A script to rotate Orca's HTML directory so that the daily plots can be archived. It is named rotate_orca_graphs. 8) Replace the small color scheme used for automatically cycling through plot colors with a longer list designed by Guy Dallaire . 9) To support configurations where it does not make sense to make all of the hourly, daily, weekly, monthly, quarterly, and yearly plots, i.e. when only one measurement per day is made and having a daily plot would not show anything, Orca now understands the global configuration file options "generate_hourly_plot", "generate_daily_plot", "generate_weekly_plot", "generate_monthly_plot", "generate_quarterly_plot", and "generate_yearly_plot". When an argument value of 0 is given to the parameter, the plot is not created. If the particular parameter is not used or the value is not 0, then the plot is created. 10) Add a new configuration file option for plots named "summary_format" which specifies the format for the summary values as passed to the RRDtool GPRINT function. The same format is used for each number within a single summary line, but you can specify multiple "summary_format"'s options if there are multiple plots on the graph. The default value, which has not changed from previous Orca versions, is '%9.3lf %S'. Patch from Alex Howansky . 11) Remove the "date_format" configuration option as it was never supported. This is replaced by a new "date_parse" configuration file parameter which supports converting arbitrary strings in the input source data files that somehow represent time into an Unix epoch time usable by Orca. The "date_parse" parameter appears in a group field and is an arbitrary Perl subroutine that is given two arguments, the first being the name of the file where the data is loaded and the second the string from the "date_source" column that contains some time information. The subroutine should return the Unix epoch time. If this option is not specified, then Orca assumes that the string holds the Unix epoch time. This Perl subroutine is only used if the file's date source is not specified to be the file's last modified time as indicated to Orca by use of the "date_source" file_mtime configuration file option. 12) Remove the "sub_dir" configuration file parameter and now always create sub directories for RRD, image and HTML files. This removes the problem when a simple Orca configuration file was used and Orca did not create subdirectories and then additional groups where added to the configuration file Orca would begin to use subdirectories and the existing RRD, image and HTML files would be in the wrong location. In this case, Orca would reload all of the input data. 13) Allow the "late_interval" configuration parameter to appear in a configuration file group. If it does not appear in a group listing, then use the global "late_interval" value. 14) Add a new configuration file option named "require" that allows the configuration file to specify the minimum required version of Orca. 15) Orca now catches SIGPIPEs so the lock directory is properly removed when Orca's STDOUT and/or STDERR is piped to a process, such as less, and the piped to process exits before Orca does. 16) Fix a bug where Orca's documentation stated that the configuration file can specify different data_min, data_max, and data_type's for each data in a plot but the code did not. Make the modifications in Orca to support this feature. 17) When loading a configuration file, now do a complete check of it for errors before quitting, instead of quitting after a single error. 18) Include and require the following package updates: Digest::MD5 2.13, RRDtool 1.0.33 and Storable 1.0.11. 19) The email address blair@akamai.com that I used no longer works, as I have left Akamai. I can be reached at blair@gps.caltech.edu. These following changes are what's new in orcallator.se 1.28 and the orcallator.cfg file since version 1.23 which was included with Orca 0.26. All of the changes below are taken advantage of in the included orcallator.cfg and start_orcallator files. 20) When orcallator.se was running on a system with an older version of SE the p_vmstat.scan variable is an integer and the sprintf to %8.3f fails, resulting in a perceived scan rate of 0 pages per second. Now always add 0.0 to p_vmstat.scan to get a double. 21) Fix a typo where nil was misspelled as nik. 22) Make sure to check the return from stat() on the web server access log in case the file is missing. Use fstat() instead of stat() when a file descriptor is available. 23) Print the portion of time running in idle mode with some process waiting for block I/O as wio% and otherwise completely idle time as idle%. Update orcallator.cfg.in to fix a bug in plotting the CPU usage plots where the idle% time was calculated by subtracting the user percent time usr% and the system percent time sys% from 100%. This did not take into account the wait on IO time wio% that the system measures and hence the idle% was overestimated. Now plot the correct idle% and the wio%. 24) Recoded measure_disk() to access the RAWDISK interface to sys_kstat device information to allow the activity on Sun's A1000 and Clariion Raid controller drives to be seen. Apparently the pseudo drivers do not update the kstat interface. It is also inverts the fix provided by version 1.23 to avoid over-counting md devices. By suppressing stats from slices and metadevices and instead reporting on full devices such as c0t0d0 or sd0. Note: This may have introduced an interaction with the live_rules.se class monitoring of drive performance. Prevent floppy disks and tape drives from RAWDISK. Added wio% to measure wait time since the idle calculation is wrong without this. Prevent filesystems mounted under /snapshots from being seen. Patch contributed by Alan LeGrand . NEW IN ORCA 0.26 ================ 1) Due to some changes at onelist.com, the name of this mailing must change. The -help at the end of orca-help is being reserved for the mailing list named "orca". As such, I'm renaming this list to orca-users@onelist.com. 2) Major speedup in starting Orca when many different source files exist and the first line of the file is used to tell Orca what columns of data there are. Now Orca caches this information in the state file so that the source data files do no have to be opened every time Orca starts up. This has a significant speed improvement when the file is compressed, since a separate process is normally spawned to read the file. 3) Reduce the number of size of the open file descriptor cache to 100 open file descriptors since the change in 2) reduces the need for as many open file descriptors. 4) A new configuration file parameter named filename_compare was added that let's Orca determine when it can flush data from loaded source files. This prevents Orca from loading in all of the data and then flushing it to the disk when the amount of new data is large. See the Orca documentation for more information. 5) Allow Orca to work with RRDtool versions newer than 1.0.7. 6) Save Orca's state more often so that if something happens to the Orca process, it does not have to reload all the data from the source data files it previously read. 7) Have Orca print the current memory usage and CPU time more often. This attempts to use the proper ps command for the operating system Orca was configured on. 8) Take all of the modules in the monolithic orca script and move them into their own files. This improves readability and maintainability. 9) Require the latest versions of Data::Dumper (2.101), Digest::MD5 (2.09), Math::IntervalSearch (1.05), RRDs (1.0.13), and Storable (0.6.9). 10) Use arrays instead of hashes as the internal structure for many objects. This should speed access to object members and make their memory footprint smaller. 11) Use a new integer file identifier (FID) instead of the filename to refer to in internal objects. There is now a global array keyed by the FID that gives the name and a hash keyed by filename that returns the FID. The FID is used where the filename previously was used. This reduces Orca's memory usage by 10%, since only one copy of a long filename is needed. These following changes are what's new in orcallator.se 1.23 and the orcallator.cfg file since version 1.20 which was included with Orca 0.25. All of the changes below are taken advantage of in the included orcallator.cfg and start_orcallator files. 12) Include code to record and plot the NFS v2 and v3 server statistics. The new statistics are: nfss_calls, the number of NFS calls to the NFS server, nfss_bad, the number of bad NFS calls per second, and v{2,3}{reads,writes}, which are nfss_calls broken down into NFS version 2 and NFS version 3 calls. The sum of v{2,3}{reads,writes} will be less than nfss_calls as the other types of NFS calls, such as getattr and lookup, are not included. Contributed by Paul Haldane . This code is enabled by the standard -DWATCH_OS or individually by -DWATCH_NFS_SERVER. The define -DWATCH_NFS has been superseded by -DWATCH_NFS_CLIENT, but to keep backwards compatibility, -DWATCH_NFS_CLIENT will be defined if -DWATCH_NFS is defined. 13) When orcallator was running on a system with DiskSuite, the same physical disk was listed multiple times when it appeared in the same metadevice. The solution to the problem is not to build the c0t0d0 name but use the long disk name provided by the long_name string. Patch contributed by Paul Haldane . 14) Prevent core dumps on extremely long access log lines. NEW IN ORCA 0.25 ================ 1) I have moved companies from Yahoo!/GeoCities to Akamai. My email address has changed to blair@akamai.com and the old one will no longer work. Also, I have moved the entire archive from www.geocities.com/~bzking/ to www.gps.caltech.edu/~blair/orca/ 2) Fix an important bug where Orca would crash if column names mentioned in the configuration file did not exist in the source data files. 3) Orca, when it received the HUP signal, will look for new source data files the next time it runs through the main loop. If you have a constantly running Orca, this is a simpler and faster solution than restarting Orca, which takes time to reread all the source files. 4) Add a new make target orcallator_run_at_boot which will install the proper files into the proper /etc/*.d/ directories to start orcallator at boot time and shut it down at halt time. 5) The previous default orcallator.cfg would not find compressed orcallator files. 6) Remove the plot configuration option "optional" which made plots optional. Now make plots optional by default and use the keyword "required" to make them required. If a plot is required, then it will always display, even if there is no data loaded for it. 7) Change the behavior of warnings when data requested to be plotted in the configuration file does not exist in the source data files. Any resulting errors from this, such as cannot compile errors, are only sent when the verbose level is above 1 or when the plot is required. 8) Include Storable 0.6.7 but continue to only require Storable 0.6.3. NEW IN ORCA 0.24 ================ 1) Installation notes. Due to the way Orca generated image and HTML files are named, read the INSTALL file. Otherwise, you will have some left over cruft that will waste disk space and you will have to reload all of your source data files. 2) Orca now runs under older Perls: Perl 5.004_01 or later. 3) Switch to generate PNGs instead of GIFs. They take up 1/3 less disk space and are created at least 10% faster. If you want Orca to generate GIFs instead of PNGs, give it the -gifs flag. 4) Read in .gz files using gunzip, .Z files using uncompress, and .bz2 files with bunzip2. 5) Add to the plot title the type of the plot (i.e. 'Yearly') in every image. 6) Add a href field for a plot. This, if defined, appends a HREF to a target name letting you point the user to a page to get more information. The default orcallator.cfg has these added to point to the new documentation web page http://www.geocities.com/~bzking/docs/orcallator.html for all orcallator.se recorded data. 7) Add a new option named base that lets the user specify either a base of 1000 or 1024 for autoscaling plots. This is useful for memory (base 1024) vs network (base 1000) measurements and is used in generating correctly calculating the base to use in calculating the Y axis. 8) The word logarithmic can now be used for a plot to create a logarithmic Y axis scale. 9) Orca no longer lists all the source files it finds to shorten verbose output. This output is now generated if the verbose level is greater than one. 10) Do not overwrite existing HTML files when creating new versions until the new version is finished. This allows people to better view existing pages until the new page is completely finished. 11) All generated HTML and image filenames are now created using a different set of mappings. Now orcallator -> o orca -> o _times_ -> _X_ _percent_ -> _pct_ _number_ -> _num_. All older installations of Orca will need to be renamed unless you want to load in all your data again. You can perform this renaming on your files by running make upgrade, or if you have directories that are not normally covered by the Orca install, run src/upgrade_installation followed by the list of directories to search in. Pass the -n flag to upgrade_installation if you want to see the renames that will be performed without actually performing them. 12) New HTML and image filenames are shorter, which can save 10% in disk space on a large installation according to du -k. Also, now plots containing arbitrary many data sources can be plotted without exceeding the maximum file lengths. 13) Add locking so that only one Orca can run on a single configuration file at one time. 14) Include and require RRDtool 1.0.7.2 and Math::Interpolate 1.05. Include Data::Dumper 2.101, Digest::MD5 2.09, Storable 0.6.5, with Orca. These following changes are what's new in orcallator.se 1.19 since version 1.7 which was included with Orca 0.23. All of the changes below are taken advantage of in the included orcallator.cfg and start_orcallator files. 15) Orcallator.se now has a web page describing the various measurements it makes. See http://www.geocities.com/~bzking/docs/orcallator.html. 16) If the environmental variable WEB_SERVER is defined, use its value of the as the name of the process to count for the number of web servers on the system. If WEB_SERVER is not defined, then count number of httpd's. 17) If the COMPRESSOR environmental variable is defined, then when a new log file is opened for a new day, the just closed log file is compressed using the COMPRESSOR command in the following manner: system(sprintf("%s %s &", COMPRESSOR, log_file) COMPRESSOR should be set to something like "gzip -9", or "compress", or "bzip2 -9". If the configure script finds both a compressor and uncompressor tool, such as both gzip and gunzip, then start_orcallator will inform orcallator.se to compress the log files at the end of a day. 18) New measurements. The first column lists the define that must be passed to the SE interpreter to obtain the measurement. If you use the WATCH_OS define, then all of these are now measured. WATCH_MUTEX - ncpus - number of CPUs on the system WATCH_CPU - #proc/s - 5 minute average process spawn rate if root WATCH_CPU - #proc/p5s - maximum 5 second process spawn rate if root WATCH_CPU - scanrate - page scan rate in pages per second WATCH_DISK - disk_runp_c\d+t\d+d\d+ - run percent for each disk WATCH_DISK - disk_rd/s - system wide read operations per second WATCH_DISK - disk_wr/s - system wide write operations per second WATCH_DISK - disk_rK/s - system wide kilobytes read per second WATCH_DISK - disk_wK/s - system wide kilobytes written per second WATCH_RAM - freememK - number of free kilobytes of memory on the system WATCH_MOUNTS - mntC_* - capacity of disk in kilobytes WATCH_MOUNTS - mntc_* - inode capacity of disk WATCH_MOUNTS - mntU_* - used capacity of disk in kilobytes WATCH_MOUNTS - mntu_* - used inode capacity of disk WATCH_MOUNTS - mntA_* - available kilobytes for non-root users WATCH_MOUNTS - mnta_* - available inodes for non-root users WATCH_MOUNTS - mntP_* - percentage of kilobytes used for non-root users WATCH_MOUNTS - mntp_* - percentage of inodes used for non-root users 19) Add some smarts so that if the number of interfaces, physical disks, or mounted partitions changes, then a new header is printed. This will prevent column name and data mixups when the system configuration changes. 20) Prevent a division by zero in calculating the mean_disk_busy if the number of disks on the system is 0. 21) Fix a bug in the disk_mean calculation where it was being divided by the wrong disk_count. Now it should be much larger and in scale with disk_peak. 22) Increase the number of characters for each network interface from four to five. 23) If WATCH_YAHOO is defined, then process the access log as a Yahoo! style access log. 24) Restructure the code to handle different web server access log formats easier. NEW IN ORCA 0.23 ================ 1) Fix two important bugs in orcallator_running and start_orcallator which where using old PERCOLLATOR_DIR variables instead of the new ORCALLATOR_DIR. NEW IN ORCA 0.22 ================ 1) Add a new command line option (-r) that tells Orca to only update the RRD data files and not to generate any HTML or GIF files. This is useful if you are loading in a large amount of data in several invocations of Orca and do not want to create the HTML and GIF files in each run since it is time consuming. 2) Fix a bug where if the number of columns changed in from one source file to the next source file in a files group, the column index used to get a particular value is the index for the old file and not the index for the new file. This required some internal restructuring the speeds Orca up slightly. 3) Add a new plot called quarterly that displays the last 100 days of data. This is a nice transition between the monthly and yearly plots. 4) A temporary fix: Skip and do not complain about input source data file lines containing the word timestamp in it. 5) Removed some unused methods to make Orca smaller. 6) Added some more documentation to orcallator.cfg. 7) Make Orca slightly faster. NEW IN ORCA 0.21 ================ 1) Every file containing the words percollator, percol and perc has been renamed to contain the word orcallator. A new make target named migrate will change all filenames in an installed Orca directory to the new names. This "make migrate" should only be run after killing any running percollator.se's and Orca processes and before installing this version of Orca. 2) Orca now includes Digest::MD5 2.07, Math::Interpolate 1.02, RRDs 0.99029, and Storable 0.603 in packages/ to make installation easier. The configure script now determines if you have the requisite version of these packages installed in your Perl and will modify the Makefiles to build the necessary modules. 3) Orca now displays the daily, weekly, monthly, yearly plots of all measurements even if there is only one group. Orca used to not make the web pages showing the different measurements all on one page if there was only one group. 4) Add GIF width and height tags to img src's using the returned GIF size from RRDs::graph to speed with page rendering. 5) Some minor HTML fixes to have the pages look cleaner for Netscape browsers. 6) Make use of RRDtool 0.99.12's legend formatting to clean up the GIF legends and to store the minimum and maximum values of a data stream. 7) Remove the LAST RRA from all Orca created RRAs since it is not required to get the current value from an RRA. This requires rebuilding all RRDs. 8) Add x86 Ethernet device (elxl*) plotting to orcallator.cfg.in. 9) Do group name substitution on the Y legend in GIFs. 10) Orca used to delete RRD files even if they are created with new versions or Orca. Now the RRD file is used if it is at least as new as the version a particular version of Orca needs. 11) Orca now dies if an unrecognized command line option is given. 12) Speed up the count_proc() routine in orcallator.se by 20%. 13) Perl_glob does not die if it cannot open a directory. It will also not follow directories named .. if they happen to match a regular expression. It also only returns files matching the regular expressions. 14) Change all references from http://www.geocities.com/ResearchTriangle/Thinktank/4996/ to http://www.geocities.com/~bzking/ 15) Speed up Orca by using Storable to do deep clones of objects instead of using Data::Dumper and eval. 16) Replace some linear search by a hash lookup to speed up Orca. 17) Fix some bugs in orcallator.se. 18) Clean up some error messages.