Hi Sandeep,

Thanks for liking! On your question… first let’s check the source code of both of the functions:

/**
* Creates a constraint that asserts on a column completion.
* Uses the given history selection strategy to retrieve historical completeness values on this
* column from the history provider.
*
*
@param column Column to run the assertion on
*
@param assertion Function that receives a double input parameter and returns a boolean
*
@param hint A hint to provide additional context why a constraint could have failed
*
@return
*/
def hasCompleteness(
column: String,
assertion: Double => Boolean,
hint: Option[String] = None)
: CheckWithLastConstraintFilterable = {
addFilterableConstraint { filter => completenessConstraint(column, assertion, filter, hint) }
}

/**
* Creates a constraint that asserts on a column completion.
*
*
@param column Column to run the assertion on
*
@param hint A hint to provide additional context why a constraint could have failed
*
@return
*/
def isComplete(column: String, hint: Option[String] = None): CheckWithLastConstraintFilterable = {
addFilterableConstraint { filter => completenessConstraint(column,
Check.IsOne, filter, hint) }
}

In hasCompleteness — we can pass the how much % of records we’re expecting to satisfy. In isComplete — it’s 100%.

Constraint suggestion for ‘valuable’: ‘valuable’ has less than 62% missing values

The corresponding scala code is .hasCompleteness(“valuable”, _ >= 0.38, Some(“It should be above 0.38!”))

Written by

Tech enthusiast, Azure Big Data Architect.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store