On-Premises Data Gateway EvaluationContext
New field spotted in July 2023 update
Prelude
On-premises Data Gateway is one of most important and (unfortunately) underestimated components in a Power BI architecture.
For some of our customers we play the Power BI Administrator role (or Fabric Administrator if you will) and one of our duties is to take care of Gateways.
Setting up a good maintenance, monitoring and governance for Gateways is important since they could easily become your single point of failure.
Gateway’s logs can give you a lot of insights over your workloads.
Spotted in July 2023
One of our customer is running a custom Power BI solution developed by us to monitor its Gateways leveraging standard Gateway logs. Every month we manage the upgrade process to the latest version available but this month our monitoring solution’s refresh crashed after the upgrade.
Data source error:{“error”:{“code”:”DM_GWPipeline_Gateway_MashupDataAccessError”,”pbi.error”:{“code”:”DM_GWPipeline_Gateway_MashupDataAccessError”,”parameters”:{},”details”:[{“code”:”DM_ErrorDetailNameCode_UnderlyingErrorCode”,”detail”:{“type”:1,”value”:”-2147467259"}},{“code”:”DM_ErrorDetailNameCode_UnderlyingErrorMessage”,”detail”:{“type”:1,”value”:”There were more columns in the result than expected.”}},{“code”:”DM_ErrorDetailNameCode_UnderlyingHResult”,”detail”:{“type”:1,”value”:”-2147467259"}},{“code”:”Microsoft.Data.Mashup.ValueError.Count”,”detail”:{“type”:1,”value”:”7"}},{“code”:”Microsoft.Data.Mashup.ValueError.Reason”,”detail”:{“type”:1,”value”:”DataFormat.Error”}}],”exceptionCulprit”:1}}} Table: Query Execution Stats.
Message is quite clear, but it wasn’t easy to identify what caused the issue just after Gateway upgrade.
We realized that Gateway application doesn’t close opened logs, even after an upgrade, and continues to write where it was. The result was our application could read correctly first half of the log file and not the second half expecting 7 columns instead of 8.
Documentation doesn't help since it doesn’t give an clue about this misterious new column. After waiting for Gateway logs rollover, the new column’s name appear in the latest QueryStartLog_*.json file.
The new column is called EvaluationContext.
Guessing its purpose
Without an official documentation we can only guess what it is meant for looking to the its content.
At a first sight it seems a JSON document that gives some insights about the query.
Among other values that can be found in other log files, WorkspaceId and SKU promise to tell us which is the workspace from which the query started and if it’s a pro or a premium workspace.
This addition to log can help us building stats on workspaces defining which sends more queries or heavier queries. This ranking can be useful also to idenfity root cause in case our Gateway is suffering for performance issue.
Conclusions
Microsoft dropped an update to On-premises Data Gateway on July 2023 that add a new field to standard QueryStartLog logging file. This field is called EvaluationContext and contains a JSON document that gives some more context to the query. Tracking queries by WorkspaceId can help identify most resource consuming.
