Skip to content

Try to optimize the poseidon loader

Arun Kumar Tharanatha requested to merge loader_optimize into master
  1. This is a OPTIONAL enhancement (not a fix) aimed at improving the loader performance. Hence the previously existing methods are not deleted.
  2. It was found that the performance improvement is not so significant. Example: On SF10 dataset, it was just faster by 2.3 minutes.
  3. The code changes are done only for "STRICT mode loading".
  4. LOGIC: The basic idea is to avoid using boost::any() and it corresponding casting. To do this, I retain the parsed input fields as std::strings until it is needed to create a p_item object. Inserting into dictionary and converting into appropriate datatypes happens just before creating a p_item() object. Thus we can avoid using boost::any() and boost::any_cast<>().
  5. Additionally, you may find some other optimizations like replacing the custom CSV parser with a simple while() loop.

Merge request reports