Relax dependency on lightgbm

[jlopezpena]

Latest version of lightgbm is 4.3, but arfs is hard-pinning version 3.3.1. This PR relaxes the dependency to allow for more recent lightgbm versions

[ThomasBury]

Hi @jlopezpena, Upgrading to LightGBM v4 won't be a straightforward process. Here's why:

Output Shape Changes: The output shapes of LightGBM models have changed across versions. We'll need to verify compatibility with our current code.

Optuna Compatibility: Compatibility testing is required to ensure Optuna works seamlessly with LightGBM v4.

Downgrading Context:
downgrading to LightGBM v3.3.1 due to compatibility issues with private packages I'm developing and which use ARFS (as referenced in the changelog) see also #32

I'll try to check in the near future if I can work on it

[jlopezpena]

Optuna works with lighgbm >= 4 since last summer: optuna/optuna#4844

Is there any test suite I can run to check if anything breaks?

[jlopezpena]

For reference, I just ran my modelling pipeline on my branch, with LightGBM 4.3.0, and it ran successfully without any errors. I am obviously not using every method in ARFS, but I did use the following:

• CollinearityThreshold
• VariableImportance with lgb_kwargs = {"objective": "xentropy", "zero_as_missing": False} (regression task)
• GrootCV with cutoff=1, objective="xentropy", nfolds=10, n_iter=10
  I am always using fastshap=False because the FastTreeShap hasn't been updated since a very long time ago, and hasn't kept up with original SHAP

[ThomasBury]

For reference, I just ran my modelling pipeline on my branch, with LightGBM 4.3.0, and it ran successfully without any errors. I am obviously not using every method in ARFS, but I did use the following:

• CollinearityThreshold
• VariableImportance with lgb_kwargs = {"objective": "xentropy", "zero_as_missing": False} (regression task)
• GrootCV with cutoff=1, objective="xentropy", nfolds=10, n_iter=10
  I am always using fastshap=False because the FastTreeShap hasn't been updated since a very long time ago, and hasn't kept up with original SHAP

Thanks for reviewing!

While I don't currently have a dedicated unit test suite, I'm still working on adapting the existing one to function with the new ARFS API.

In the meantime, the provided tutorial notebooks act as an integration test suite. I'll be running these tests with the relaxed version of LightGBM. Expect to see an update on the unit test suite by the end of the month.

You're right, fastshap is a nice project but isn't yet ready for production purposes.