You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As shown during the .NET Overview Session today at BUILD and as discussed with the ML Team there is only a text based reader for training data. Having a binary reader for training data would be highly beneficial to my use case.
Jarred,
(@jcapellman) that's an interesting suggestion. Are you looking at custom binary deserialization reader?
There are couple of options for you. One is erformant, but complex, and another is slower, but simpler.
If you are looking for ultimate performance, you will need to implement a IDataLoader interface. This will be a loader component that returns back a special datatype, IDataView. you can use the ParquetLoader as a template.
Alternatively, you can use your existing code that deserializes a binary file and make it return an IEnumerable< T > where T is the type that carries your features to the LearningPipeline. you can look at the HousePriceData example to see what T should look like. Let us know if you have any sparse data, as we have special methods to handle that as well
DRI RESPONSE: This is something what we want to have. But it doesn't look like a major priority for us right now, considering what we allow user to use enumeration of classes(List or Array) as data source.
Sorry for not looping back on this earlier - been crazy time for me since BUILD.
As we keep evaluating ML.NET vs XGBoost, one of the biggest things is our training data is now over 300 million files to do feature extraction and then train a model. As @codemzs mentioned these files are binary to begin with and keeping it in binary would save a ton of space over having 24k features written to a text file like we do today for those 300 million files.
@codemzs
Thank you - on a side note, really appreciate you, the ML.NET and the community for making this an incredible framework to work with. Transformed my side projects and am going to use it for my DSc starting in August.
Activity
markusweimer commentedon May 8, 2018
Which file format do you have in mind for us to support?
codemzs commentedon May 8, 2018
We met him at BUILD and his data is in raw binary format, the reason being he has several thousand features, so may be to keep the dataset compact.
glebuk commentedon May 8, 2018
Jarred,
(@jcapellman) that's an interesting suggestion. Are you looking at custom binary deserialization reader?
There are couple of options for you. One is erformant, but complex, and another is slower, but simpler.
If you are looking for ultimate performance, you will need to implement a IDataLoader interface. This will be a loader component that returns back a special datatype, IDataView. you can use the ParquetLoader as a template.
Alternatively, you can use your existing code that deserializes a binary file and make it return an IEnumerable< T > where T is the type that carries your features to the LearningPipeline. you can look at the HousePriceData example to see what T should look like. Let us know if you have any sparse data, as we have special methods to handle that as well
abgoswam commentedon Oct 15, 2018
DRI RESPONSE: This is something what we want to have. But it doesn't look like a major priority for us right now, considering what we allow user to use enumeration of classes(List or Array) as data source.
jcapellman commentedon Oct 16, 2018
Sorry for not looping back on this earlier - been crazy time for me since BUILD.
As we keep evaluating ML.NET vs XGBoost, one of the biggest things is our training data is now over 300 million files to do feature extraction and then train a model. As @codemzs mentioned these files are binary to begin with and keeping it in binary would save a ton of space over having 24k features written to a text file like we do today for those 300 million files.
jwood803 commentedon Mar 2, 2019
Would the addition of the binary saving and loading PR help with this?
jcapellman commentedon Mar 3, 2019
@jwood803
Yes it would
Use better AutiFit overload and ignore Multiclass (dotnet#64)
code migration from machinelearning-automl to machinelearning repo fe…
codemzs commentedon Jun 30, 2019
Closing this issue as we already have this functionality here: https://github.com/dotnet/machinelearning/tree/master/src/Microsoft.ML.Data/DataLoadSave/Binary
jcapellman commentedon Jun 30, 2019
@codemzs
Thank you - on a side note, really appreciate you, the ML.NET and the community for making this an incredible framework to work with. Transformed my side projects and am going to use it for my DSc starting in August.
Use better AutiFit overload and ignore Multiclass (dotnet#64)
Added onnx export functionality for LpNormNormalizingTransformer (#4161)
Added Onnx export functionality to PCATransformer (#4188)