Defining Dataset

Learn how to define a Dataset.

val df:DataFrame = framework.dataAccess
  .entity(
    new EntityDatasetBuilder()
      .ofType("configuration/entityTypes/HCP")
      .select("Id")
      .select("attributes")
      .explode("attributes.FirstName","FirstName")
      .explode("attributes.LastName","LastName")
      .explode("attributes.Gender","Gender")
      .explode("attributes.Specialities.SpecialtyType","SpecialtyType")
      .explode("attributes.Specialities.Specialty", "Specialty")
      .asTable("hcps"))
  .build()
  df.cache()

Output:

df: org.apache.spark.sql.DataFrame = 
[Id: bigint, attributes: struct<FirstName:array<string>,LastName:array<string>,Gender:array<string>,Specialities:array<struct<SpecialtyType:array<string>,Specialty:array<string>>>>, 
FirstName: string, LastName: string, Gender: string, SpecialtyType: string, Specialty: string]