Combining BERT + RGCNs is great. Transformers on text + categorical features, then GNNs for learning over connected data, esp their choice of RGCNs for heterogeneous ones.
Some of my favorite use cases to do here are in entity resolution / data cleaning during document ingest, mining social media interactions, and analyzing financial data (loan risk, ...). They all used to largely follow the flow here, just add feature engineering and classification decisions specific to the problem at hand. Especially for scale & automation scenarios where quality matters, this stuff helps.
How I think about this space has changed significantly with modern transformers compared to the BERT-era ones here. The paper feels closer to what we (and others) were doing before GPT4 came out. Now that LLMs can 'reason', not just embed, a lot more has opened up during the feature extraction, learning, and deciding phases. Basically pick up any new KG paper using LLMs, there is a lot to keep up with.
Happy to chat if folks are doing fun things here. We are always looking for good projects in this space as there is nuance and esp with LLMs changing so much. Exciting times!
I came back to to this post several times today to see the comments on this paper. I was curious what is significant about this paper given that it stays on Hacker News front page for hours.
Another curiosity is, what is the typical cost and GPU hours to train the model with these algorithms?
This extraction of graph structure is the "holy grail" of NLP in that it can break documents down into facts so that, say, you can store them in a database and query them in a more accurate and efficient way.
Also these science papers frequently have a very low comments to vote ratio compared to, say, articles about cars or the housing supply in California.
Some of my favorite use cases to do here are in entity resolution / data cleaning during document ingest, mining social media interactions, and analyzing financial data (loan risk, ...). They all used to largely follow the flow here, just add feature engineering and classification decisions specific to the problem at hand. Especially for scale & automation scenarios where quality matters, this stuff helps.
How I think about this space has changed significantly with modern transformers compared to the BERT-era ones here. The paper feels closer to what we (and others) were doing before GPT4 came out. Now that LLMs can 'reason', not just embed, a lot more has opened up during the feature extraction, learning, and deciding phases. Basically pick up any new KG paper using LLMs, there is a lot to keep up with.
Happy to chat if folks are doing fun things here. We are always looking for good projects in this space as there is nuance and esp with LLMs changing so much. Exciting times!
Another curiosity is, what is the typical cost and GPU hours to train the model with these algorithms?
Also these science papers frequently have a very low comments to vote ratio compared to, say, articles about cars or the housing supply in California.