Usage: xdmod-ondemand-ingestor

Get Help
1

How do people implement ingesting the ondemand logs?

From the help, we require:
xdmod-ondemand-ingestor -r -u -d
Resource is straightforward.

From using the --dry-run flag, I’ve deduced that

  • can just be essentially a (short-hand or fqdn) name for the ondemand server
  • needs to be a local directory, i’m trying /tmp

Result of ‘xdmod-ondemand-ingestor -r ondemand -u ondemand01 -d /tmp’
was to just create the modw_ondemand data structures, but not to ‘ingest’ any data from the ondemand server.

I’m sure that i’m missing crucial details. I’m hoping to pull together some notes to offer for enhancing the existing documentation around the xdmod/ondemand integration.

Thanks

2

Documentation seems to indicate -d is the input to the program, whereas you seem to be using it as the output location? I think it’s meant to be where your apache logs are - likely /var/log/httpd.

3

Yeah - a bit confusing. xdmod-* commands run on the xdmod server. The ‘-d’ is required for ‘xdmod-ondemand-ingestor’, so the question of what people have been using for ingesting the ondemand logs remains open.

I thought the -u worked together with -d , and so indicated /var/log/ondemand-nginx, and the result was ‘invalid path’ (approximately, can’t remember details).

Time will tell. Thanks for thinking about it, Jeff

4

xdmod-ondemand-ingestor reads log files from the filesystem on the Open XDMoD web server. If this is a different web server than the Open OnDemand web server, then you’ll need to set up an intermediate process to copy the log files over before they are ingested. The -u option was removed in Open XDMoD 11.0. In prior versions, the -u option wasn’t used for transferring data; it was used for matching the value of the Referer header in the logs files, and only matching lines would be ingested. I will note that Open XDMoD 11.0 made some substantial changes/fixes to how Open OnDemand data are ingested and categorized; these are detailed in the 11.0 upgrade guide.

5

Aaron – Thanks very much. I misunderstood the documentation (still working with v10.5, as you can tell).

To be clear, are we talking about copying /var/log/ondemand-nginx// paths from ondemand server to xdmod server?

And whilest I continue to struggle with authentication, should I abandon v10.5, and upgrade to v11.0, and start again with the authentication?

Thanks

6

xdmod-ondemand-ingestor ingests Apache access logs (e.g., from /var/log/httpd/).

I would recommend backing up your work on the authentication and then upgrading to Open XDMoD 11.0. The upgrade guide is here.

7

My upgrade seems to have failed to update the database schema:

May 9 02:00:01 xdmod-vm8a CROND[3791718]: (root) CMD (/usr/bin/xdmod-ingestor --quiet)
May 9 02:00:01 xdmod-vm8a CROND[3791717]: (root) CMDOUT (SQLSTATE[42S02]: Base table or view not found: 1146 Table ‘mod_logger.log_id_seq’ doesn’t exist)

I can’t understand from xdmod docs just how to coax the database to have the new schema that comes with v11.0.1.
Do you understand how to resolve this, Aaron? ccr support has not yet responded to my queries for help in the past few days.

8

when attempting to run ‘xdmod-upgrade’ after verifying mariadb config, the same error is issued. I should have shared that, rather than the output from the nightly cron job.

[root@xdmod-vm8a httpd]# xdmod-upgrade
SQLSTATE[42S02]: Base table or view not found: 1146 Table ‘mod_logger.log_id_seq’ doesn’t exist
#0 /usr/share/xdmod/classes/CCR/DB/PDODB.php(185): PDOStatement->execute(Array)
#1 /usr/share/xdmod/classes/CCR/CCRDBHandler.php(100): CCR\DB\PDODB->execute(‘UPDATE mod_logg…’)
#2 /usr/share/xdmod/classes/CCR/CCRDBHandler.php(79): CCR\CCRDBHandler->getNextId()
#3 /usr/share/xdmod/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(39): CCR\CCRDBHandler->write(Array)
#4 /usr/share/xdmod/vendor/monolog/monolog/src/Monolog/Logger.php(344): Monolog\Handler\AbstractProcessingHandler->handle(Array)
#5 /usr/share/xdmod/classes/CCR/Logger.php(37): Monolog\Logger->addRecord(300, ‘{“message”:"No …’, Array)
#6 /usr/share/xdmod/vendor/monolog/monolog/src/Monolog/Logger.php(684): CCR\Logger->addRecord(300, ‘No upgrades nec…’, Array)
#7 /usr/bin/xdmod-upgrade(156): Monolog\Logger->warning(‘No upgrades nec…’)
#8 /usr/bin/xdmod-upgrade(46): main()
#9 {main}

9

After checking the database directly, and observing the table identified as missing:
MariaDB [mod_logger]> show tables;
±---------------------+
| Tables_in_mod_logger |
±---------------------+
| log_id_seq |
| log_level |
| log_table |
±---------------------+
3 rows in set (0.000 sec)

I restarted the mariadb service, and then ran /usr/bin/xdmod-ingestor. The output to the terminal suggests that the v11 updates to the schema now seem to be recognized – I’m not sure how to confirm that programmatically…

The page does now load – yay. No data is recognized, and appears that re-ingesting the data is necessary, at least at some point.

I shall now return my attention to auth via SSO. Our institution actually no longer supports the direct ldap approach that was recommended in recent engagement with xdmod support.

Cheers
~ Em